How to Fix Cached Schema Registry lookup causing terrible performance - java

Edit: I discovered this other question from a few years back (How to populate the cache in CachedSchemaRegistryClient without making a call to register a new schema?). It mentions that the CachedSchemaRegistryClient needs to register the schema to the actual registry to make it cached, and there has been no solution yet to work around this. So leaving my question here, but wanted that to be made aware as well.
I am working on a program that is pulling a byte array from kafka, decrypting it (so it is secure while on kafka), converting the bytes to a string, the json string to json object, looking up the schema from the schema registry (utilizing CachedSchemaRegistryClient), converting the json bytes to a generic record using the schema from the retrieved schema from the registry metadata, and then serializing that generic record into avro bytes.
After running some tests it seems that the CachedSchemaRegistyClient is the major performance drain. But from what I can tell this is the best way to go about getting the schema metadata. Have I implemented something poorly or is there some other way that this can be done that works with my use case?
Here is the code for what handles everything after the decrypting:
package org.apache.flink;
import avro.fullNested.FinalMessage;
import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient;
import io.confluent.kafka.schemaregistry.client.SchemaMetadata;
import org.apache.avro.Schema;
import org.apache.avro.generic.GenericRecord;
import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.util.Collector;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import serializers.AvroFinishedMessageSerializer;
import tech.allegro.schema.json2avro.converter.JsonAvroConverter;
public class JsonToAvroBytesParser implements FlatMapFunction<String, byte[]> {
private transient CachedSchemaRegistryClient schemaRegistryClient;
private transient AvroFinishedMessageSerializer avroFinishedMessageSerializer;
private String schemaUrl;
private Integer identityMaxCount;
public JsonToAvroBytesParser(String passedSchemaUrl, int passedImc){
schemaUrl = passedSchemaUrl;
identityMaxCount = passedImc;
}
private void ensureInitialized() {
if (schemaUrl.equals("")) {
schemaUrl = "https://myschemaurl.com/";
}
if(identityMaxCount == null){
identityMaxCount = 5;
}
if(schemaRegistryClient == null){
schemaRegistryClient = new CachedSchemaRegistryClient(schemaUrl, identityMaxCount);
}
if(avroFinalMessageSerializer == null){
avroFinalMessageSerializer = new AvroFinalMessageSerializer(FinalMessage.class);
}
}
#Override
public void flatMap(String s, Collector<byte[]> collector) throws Exception {
ensureInitialized();
Object obj = new JSONParser().parse(s);
JSONObject jsonObject = (JSONObject) obj;
try {
String headers = jsonObject.get("headers").toString();
JSONObject body = (JSONObject) jsonObject.get("requestBody");
if(headers != null && body != null){
String kafkaTopicFromHeaders = "hard_coded_name-value";
//NOTE: this schema lookup has serious performance issues.
SchemaMetadata schemaMetadata = schemaRegistryClient.getLatestSchemaMetadata(kafkaTopicFromHeaders);
//TODO: need to implement recovery method if schema cannot be reached.
JsonAvroConverter converter = new JsonAvroConverter();
GenericRecord specificRecord = converter.convertToGenericDataRecord(body.toJSONString().getBytes(), new Schema.Parser().parse(schemaMetadata.getSchema()));
byte[] bytesToReturn = avroFinishedMessageSerializer.serializeWithSchemaId(schemaMetadata, specificRecord);
collector.collect(bytesToReturn);
}
else {
System.out.println("json is incorrect.");
}
} catch (Exception e){
System.out.println("json conversion exception caught");
}
}
}
Thanks for any help in advance!

It appears the getLatestSchemaMetadata method does not use the cache. If you want your calls to use the cache to improve performance perhaps you can reorganize your program to use one of the other methods that does use the cache, perhaps lookup schema by ID or register schema by name with definition string.
I'm having trouble locating the documentation for Java (or Python, or C++) that confirms this is how SchemaRegistry works (tried here). But the .Net docs says at least in that client API the getLatest method is not cached.

Related

populate a jtable with json string results from a RESTful web service using java

I have a web service that returns JSON strings one by one based on query inputs, a GET request to the service returns this (there's only one entry in the database)
[{"checked":false,"dateposted":"2014-10-23T00:00:00","itemnumber":1,"quantity":5,"stockcheckid":1}]
at the moment I just have this System.out.println in a while loop.
what I want to do is be able to access these results in a way that I can input them into a jtable to display on a client application. I've read some guides about reading from JSON files etc but I cant find anything specific to REST web services. I hear GSON mentioned a lot, i tried that but i cant quite work out how to make it work in this context
I should also mention that the service can also send this data in XML format.
Do I somehow create a JSON file appending each new entry to it? and then populate the table from that file?
anyway here's the code that initiates the GET request.
public static void getRequest(String dataGet) {
try {
URL url = new URL("http://localhost:8080/nXXXXXXXXc/webresources/entities.stockchecks/" + dataGet);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Accept", "application/json");
if (conn.getResponseCode() != 200) {
throw new RuntimeException("Failed : HTTP error code : "
+ conn.getResponseCode());
}
BufferedReader br = new BufferedReader(new InputStreamReader(
(conn.getInputStream())));
String output;
while ((output = br.readLine()) != null) {
System.out.println(output);
}
conn.disconnect();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}`
The response is a map. You can serialise a JSON map to a Java map with Jackson like this:
import static org.junit.Assert.assertEquals;
import java.io.IOException;
import java.util.Map;
import java.util.TreeMap;
import org.junit.Test;
import com.fasterxml.jackson.annotation.JsonAnyGetter;
import com.fasterxml.jackson.annotation.JsonAnySetter;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
public class JSONtoMap {
public static final String json = "{\"B\":\"b\",\"C\":\"c\"}";
public static class POJO{
private Map<String,String> map = new TreeMap<String,String>();
#JsonAnyGetter
public Map<String, String> get() {
return map;
}
#JsonAnySetter
public void set(String name, String value) {
map.put(name, value);
}
}
#Test
public final void test() throws JsonProcessingException, IOException {
ObjectMapper jsonmapper = new ObjectMapper();
POJO p = jsonmapper.readValue(json, POJO.class);
assertEquals(jsonmapper.writeValueAsString(p),json);
}
}
I imagine you could do something similar with GSON. Another option is if you know the structure of the JSON object - in that case you make a simple POJO version and de-serialise into that instead of something like the POJO class I've defined above.
More detail and a similar version that supports XML as well as JSON mapping
Irrespective of how you obtain the data, use it to construct a suitable TableModel and use that model to construct the JTable. In this example, the model accesses a Map<String, String> to fulfill the TableModel contract; you can substitute the Map obtained using the approach shown here. Because loading the data may take an indeterminate amount of time, use a SwingWorker, as shown here.
This is simply a combination of trashgod's and tom's answer, with an example, using Jackson and a TableModel. I really just want to give camickr's BeanTableModel/RowTableModel (which is a generic class to help us easily create table models to from pojos) a try (seems to work great).
For more information/details please see this post
Entity class (properties mapped to the keys in your json)
public class Entity {
// field/property names must match to your json keys (with some exceptions)
// other wise we are required to use further annotations
private boolean checked;
private Date dateposted;
private int itemnumber;
private int quantity;
private int stockcheckid;
/*** ----- DO NOT FORGET GETTERS AND SETTERS ---- ***/
}
Main class. Note the use BeanTableModel. You will need to download this class along with RowTableModel from the link above.
public class JsonTableDemo {
public static void main(String[] args) throws Exception {
ObjectMapper mapper = new ObjectMapper();
String json = "[{\"checked\":false,\"dateposted\":\"2014-10-23T00:00:00\",\"itemnumber\":1,\"quantity\":5,\"stockcheckid\":1}]";
List<Entity> response = mapper.readValue(json,
TypeFactory.defaultInstance().constructCollectionType(
List.class, Entity.class));
RowTableModel model = new BeanTableModel<>(Entity.class, response);
JTable table = new JTable(model) {
#Override
public Dimension getPreferredScrollableViewportSize() {
return getPreferredSize();
}
};
JOptionPane.showMessageDialog(null, new JScrollPane(table));
}
}
Result
Note, for long running task (many io task included), such as requesting a rest resource, you should use a SwingWorker as explained by trashgod. When the repsonse comes in, you can basically addRow to the RowTableModel instance. For instance if we use the same json response and model as above, we could simply do something like
response = mapper.readValue(json,
TypeFactory.defaultInstance().constructCollectionType(
List.class, Entity.class));
for (Entity entity : response) {
model.addRow(entity);
}
UPDATE
I should also mention that the service can also send this data in XML format.
Looking at your json, it is an array of objects. With XML, the format is a little different, as you must have a root document element. So you can't just have say
<entity>
<prop></prop>
</entity>
<entity>
<prop></prop>
</entity>
It would need to be something like
<entities>
<entity>
<prop></prop>
</entity>
<entity>
<prop></prop>
</entity>
</entities>
That being said, using data binding, the easiest way would be to create another class to wrap a List<Entity>. Now I'm not too familiar with Jackson's XML capabilities/features, but using JAXB, you can have a class like:
#XmlRootElement
#XmlAccessorType(XmlAccessType.FIELD)
public class Entities {
#XmlElement(name = "entity")
protected List<Entity> entities;
public List<Entity> getEntities() {
if (entities == null) {
entities = new ArrayList<>();
}
return entities;
}
public void setEntities(List<Entity> entities) {
this.entities = entities;
}
}
Then you can unmarshal the below XMl into the Entities class. Here's an update demo to show both json and xml
public class JsonTableDemo {
public static void main(String[] args) throws Exception {
ObjectMapper mapper = new ObjectMapper();
String json = "[{\"checked\":false,\"dateposted\":\"2014-10-23T00:00:00\",\"itemnumber\":1,\"quantity\":5,\"stockcheckid\":1}]";
List<Entity> response = mapper.readValue(json,
TypeFactory.defaultInstance().constructCollectionType(
List.class, Entity.class));
RowTableModel jsonModel = new BeanTableModel<>(Entity.class, response);
JTable jsonTable = new JTable(jsonModel) {
#Override
public Dimension getPreferredScrollableViewportSize() {
return getPreferredSize();
}
};
String xml = "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\n"
+ "<entities>\n"
+ " <entity>\n"
+ " <checked>false</checked>\n"
+ " <dateposted>2014-10-22T17:00:00-07:00</dateposted>\n"
+ " <itemnumber>1</itemnumber>\n"
+ " <quantity>5</quantity>\n"
+ " <stockcheckid>1</stockcheckid>\n"
+ " </entity>\n"
+ "</entities>";
JAXBContext context = JAXBContext.newInstance(Entities.class);
Unmarshaller unmarshaller = context.createUnmarshaller();
Entities entities = (Entities)unmarshaller.unmarshal(new StringReader(xml));
RowTableModel<Entity> xmlModel = new BeanTableModel<>(
Entity.class, entities.getEntities());
JTable xmlTable = new JTable(xmlModel){
#Override
public Dimension getPreferredScrollableViewportSize() {
return getPreferredSize();
}
};
JPanel panel = new JPanel(new GridLayout(0, 1));
JPanel jsonPanel = new JPanel(new BorderLayout());
jsonPanel.add(new JLabel("JSON Table", SwingConstants.CENTER), BorderLayout.PAGE_START);
jsonPanel.add(new JScrollPane(jsonTable));
panel.add(jsonPanel);
JPanel xmlPanel = new JPanel(new BorderLayout());
xmlPanel.add(new JLabel("XML Table", SwingConstants.CENTER), BorderLayout.PAGE_START);
xmlPanel.add(new JScrollPane(xmlTable));
panel.add(xmlPanel);
JOptionPane.showMessageDialog(null, new JScrollPane(panel));
}
}
See more about using JAXB

How to implement auto suggest using Lucene's new AnalyzingInfixSuggester API?

I am a greenhand on Lucene, and I want to implement auto suggest, just like google, when I input a character like 'G', it would give me a list, you can try your self.
I have searched on the whole net.
Nobody has done this , and it gives us some new tools in package suggest
But i need an example to tell me how to do that
Is there anyone can help ?
I'll give you a pretty complete example that shows you how to use AnalyzingInfixSuggester. In this example we'll pretend that we're Amazon, and we want to autocomplete a product search field. We'll take advantage of features of the Lucene suggestion system to implement the following:
Ranked results: We will suggest the most popular matching products first.
Region-restricted results: We will only suggest products that we sell in the customer's country.
Product photos: We will store product photo URLs in the suggestion index so we can display them in the search results, without having to do an additional database lookup.
First I'll define a simple class to hold information about a product in Product.java:
import java.util.Set;
class Product implements java.io.Serializable
{
String name;
String image;
String[] regions;
int numberSold;
public Product(String name, String image, String[] regions,
int numberSold) {
this.name = name;
this.image = image;
this.regions = regions;
this.numberSold = numberSold;
}
}
To index records in with the AnalyzingInfixSuggester's build method you need to pass it an object that implements the org.apache.lucene.search.suggest.InputIterator interface. An InputIterator gives access to the key, contexts, payload and weight for each record.
The key is the text you actually want to search on and autocomplete against. In our example, it will be the name of the product.
The contexts are a set of additional, arbitrary data that you can use to filter records against. In our example, the contexts are the set of ISO codes for the countries we will ship a particular product to.
The payload is additional arbitrary data you want to store in the index for the record. In this example, we will actually serialize each Product instance and store the resulting bytes as the payload. Then when we later do lookups, we can deserialize the payload and access information in the product instance like the image URL.
The weight is used to order suggestion results; results with a higher weight are returned first. We'll use the number of sales for a given product as its weight.
Here's the contents of ProductIterator.java:
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.ObjectOutputStream;
import java.io.UnsupportedEncodingException;
import java.util.Comparator;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Set;
import org.apache.lucene.search.suggest.InputIterator;
import org.apache.lucene.util.BytesRef;
class ProductIterator implements InputIterator
{
private Iterator<Product> productIterator;
private Product currentProduct;
ProductIterator(Iterator<Product> productIterator) {
this.productIterator = productIterator;
}
public boolean hasContexts() {
return true;
}
public boolean hasPayloads() {
return true;
}
public Comparator<BytesRef> getComparator() {
return null;
}
// This method needs to return the key for the record; this is the
// text we'll be autocompleting against.
public BytesRef next() {
if (productIterator.hasNext()) {
currentProduct = productIterator.next();
try {
return new BytesRef(currentProduct.name.getBytes("UTF8"));
} catch (UnsupportedEncodingException e) {
throw new Error("Couldn't convert to UTF-8");
}
} else {
return null;
}
}
// This method returns the payload for the record, which is
// additional data that can be associated with a record and
// returned when we do suggestion lookups. In this example the
// payload is a serialized Java object representing our product.
public BytesRef payload() {
try {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutputStream out = new ObjectOutputStream(bos);
out.writeObject(currentProduct);
out.close();
return new BytesRef(bos.toByteArray());
} catch (IOException e) {
throw new Error("Well that's unfortunate.");
}
}
// This method returns the contexts for the record, which we can
// use to restrict suggestions. In this example we use the
// regions in which a product is sold.
public Set<BytesRef> contexts() {
try {
Set<BytesRef> regions = new HashSet();
for (String region : currentProduct.regions) {
regions.add(new BytesRef(region.getBytes("UTF8")));
}
return regions;
} catch (UnsupportedEncodingException e) {
throw new Error("Couldn't convert to UTF-8");
}
}
// This method helps us order our suggestions. In this example we
// use the number of products of this type that we've sold.
public long weight() {
return currentProduct.numberSold;
}
}
In our driver program, we will do the following things:
Create an index directory in RAM.
Create a StandardTokenizer.
Create an AnalyzingInfixSuggester using the RAM directory and tokenizer.
Index a number of products using ProductIterator.
Print the results of some sample lookups.
Here's the driver program, SuggestProducts.java:
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.UnsupportedEncodingException;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.search.suggest.analyzing.AnalyzingInfixSuggester;
import org.apache.lucene.search.suggest.Lookup;
import org.apache.lucene.store.RAMDirectory;
import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.Version;
public class SuggestProducts
{
// Get suggestions given a prefix and a region.
private static void lookup(AnalyzingInfixSuggester suggester, String name,
String region) {
try {
List<Lookup.LookupResult> results;
HashSet<BytesRef> contexts = new HashSet<BytesRef>();
contexts.add(new BytesRef(region.getBytes("UTF8")));
// Do the actual lookup. We ask for the top 2 results.
results = suggester.lookup(name, contexts, 2, true, false);
System.out.println("-- \"" + name + "\" (" + region + "):");
for (Lookup.LookupResult result : results) {
System.out.println(result.key);
Product p = getProduct(result);
if (p != null) {
System.out.println(" image: " + p.image);
System.out.println(" # sold: " + p.numberSold);
}
}
} catch (IOException e) {
System.err.println("Error");
}
}
// Deserialize a Product from a LookupResult payload.
private static Product getProduct(Lookup.LookupResult result)
{
try {
BytesRef payload = result.payload;
if (payload != null) {
ByteArrayInputStream bis = new ByteArrayInputStream(payload.bytes);
ObjectInputStream in = new ObjectInputStream(bis);
Product p = (Product) in.readObject();
return p;
} else {
return null;
}
} catch (IOException|ClassNotFoundException e) {
throw new Error("Could not decode payload :(");
}
}
public static void main(String[] args) {
try {
RAMDirectory index_dir = new RAMDirectory();
StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_48);
AnalyzingInfixSuggester suggester = new AnalyzingInfixSuggester(
Version.LUCENE_48, index_dir, analyzer);
// Create our list of products.
ArrayList<Product> products = new ArrayList<Product>();
products.add(
new Product(
"Electric Guitar",
"http://images.example/electric-guitar.jpg",
new String[]{"US", "CA"},
100));
products.add(
new Product(
"Electric Train",
"http://images.example/train.jpg",
new String[]{"US", "CA"},
100));
products.add(
new Product(
"Acoustic Guitar",
"http://images.example/acoustic-guitar.jpg",
new String[]{"US", "ZA"},
80));
products.add(
new Product(
"Guarana Soda",
"http://images.example/soda.jpg",
new String[]{"ZA", "IE"},
130));
// Index the products with the suggester.
suggester.build(new ProductIterator(products.iterator()));
// Do some example lookups.
lookup(suggester, "Gu", "US");
lookup(suggester, "Gu", "ZA");
lookup(suggester, "Gui", "CA");
lookup(suggester, "Electric guit", "US");
} catch (IOException e) {
System.err.println("Error!");
}
}
}
And here is the output from the driver program:
-- "Gu" (US):
Electric Guitar
image: http://images.example/electric-guitar.jpg
# sold: 100
Acoustic Guitar
image: http://images.example/acoustic-guitar.jpg
# sold: 80
-- "Gu" (ZA):
Guarana Soda
image: http://images.example/soda.jpg
# sold: 130
Acoustic Guitar
image: http://images.example/acoustic-guitar.jpg
# sold: 80
-- "Gui" (CA):
Electric Guitar
image: http://images.example/electric-guitar.jpg
# sold: 100
-- "Electric guit" (US):
Electric Guitar
image: http://images.example/electric-guitar.jpg
# sold: 100
Appendix
There's a way to avoid writing a full InputIterator that you might find easier. You can write a stub InputIterator that returns null from its next, payload and contexts methods. Pass an instance of it to AnalyzingInfixSuggester's build method:
suggester.build(new ProductIterator(new ArrayList<Product>().iterator()));
Then for each item you want to index, call the AnalyzingInfixSuggester add method:
suggester.add(text, contexts, weight, payload)
After you've indexed everything, call refresh:
suggester.refresh();
If you're indexing large amounts of data, it's possible to significantly speedup indexing using this method with multiple threads: Call build, then use multiple threads to add items, then finally call refresh.
[Edited 2015-04-23 to demonstrate deserializing info from the LookupResult payload.]

JWT (JSON Web Token) library for Java [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 3 years ago.
Improve this question
I am working on a web application developed using Java and AngularJS and chose to implement token authentication and authorization.
For the exercise purpose, I've come to the point where I send the credentials to the server, generate a random token store it and send it back to the client.
At every request to the server I'm attaching the token in the header and it works perfectly.
For the authentication point of view is perfect and wouldn't need more.
However, I now want to keep track of the user type (admin, regular user...), as well as it's id, or any other unique field; as I understood I have to encrypt that in the token that I'm sending back to the client during the log in action. Is that correct?
Is there any JWT library that you used and can generate, encrypt and decrypt such tokens?
A link to the library's API and Maven dependency would be much appreciated.
Thanks
JJWT aims to be the easiest to use and understand JWT library for the JVM and Android:
https://github.com/jwtk/jjwt
If anyone in the need for an answer,
I used this library: http://connect2id.com/products/nimbus-jose-jwt
Maven here: http://mvnrepository.com/artifact/com.nimbusds/nimbus-jose-jwt/2.10.1
By referring to https://jwt.io/ you can find jwt implementations in many languages including java. Also the site provide some comparison between these implementation (the algorithms they support and ....).
For java these are mentioned libraries:
https://github.com/jwtk/jjwt
https://github.com/auth0/java-jwt ( A tutorial at https://auth0.com/docs/server-apis/java)
https://bitbucket.org/b_c/jose4j
https://bitbucket.org/connect2id/nimbus-jose-jwt
This library seems to work well: https://code.google.com/p/jsontoken/ .
It depends on Google Guava. Here are the Maven artifacts:
<dependency>
<groupId>com.googlecode.jsontoken</groupId>
<artifactId>jsontoken</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>18.0</version>
</dependency>
The library is in fact used by Google Wallet.
Here is how to create a jwt, and to verify it and deserialize it:
import java.security.InvalidKeyException;
import java.security.SignatureException;
import java.util.Calendar;
import java.util.List;
import net.oauth.jsontoken.JsonToken;
import net.oauth.jsontoken.JsonTokenParser;
import net.oauth.jsontoken.crypto.HmacSHA256Signer;
import net.oauth.jsontoken.crypto.HmacSHA256Verifier;
import net.oauth.jsontoken.crypto.SignatureAlgorithm;
import net.oauth.jsontoken.crypto.Verifier;
import net.oauth.jsontoken.discovery.VerifierProvider;
import net.oauth.jsontoken.discovery.VerifierProviders;
import org.apache.commons.lang3.StringUtils;
import org.bson.types.ObjectId;
import org.joda.time.DateTime;
import com.google.common.collect.Lists;
import com.google.gson.JsonObject;
/**
* Provides static methods for creating and verifying access tokens and such.
* #author davidm
*
*/
public class AuthHelper {
private static final String AUDIENCE = "NotReallyImportant";
private static final String ISSUER = "YourCompanyOrAppNameHere";
private static final String SIGNING_KEY = "LongAndHardToGuessValueWithSpecialCharacters#^($%*$%";
/**
* Creates a json web token which is a digitally signed token that contains a payload (e.g. userId to identify
* the user). The signing key is secret. That ensures that the token is authentic and has not been modified.
* Using a jwt eliminates the need to store authentication session information in a database.
* #param userId
* #param durationDays
* #return
*/
public static String createJsonWebToken(String userId, Long durationDays) {
//Current time and signing algorithm
Calendar cal = Calendar.getInstance();
HmacSHA256Signer signer;
try {
signer = new HmacSHA256Signer(ISSUER, null, SIGNING_KEY.getBytes());
} catch (InvalidKeyException e) {
throw new RuntimeException(e);
}
//Configure JSON token
JsonToken token = new net.oauth.jsontoken.JsonToken(signer);
token.setAudience(AUDIENCE);
token.setIssuedAt(new org.joda.time.Instant(cal.getTimeInMillis()));
token.setExpiration(new org.joda.time.Instant(cal.getTimeInMillis() + 1000L * 60L * 60L * 24L * durationDays));
//Configure request object, which provides information of the item
JsonObject request = new JsonObject();
request.addProperty("userId", userId);
JsonObject payload = token.getPayloadAsJsonObject();
payload.add("info", request);
try {
return token.serializeAndSign();
} catch (SignatureException e) {
throw new RuntimeException(e);
}
}
/**
* Verifies a json web token's validity and extracts the user id and other information from it.
* #param token
* #return
* #throws SignatureException
* #throws InvalidKeyException
*/
public static TokenInfo verifyToken(String token)
{
try {
final Verifier hmacVerifier = new HmacSHA256Verifier(SIGNING_KEY.getBytes());
VerifierProvider hmacLocator = new VerifierProvider() {
#Override
public List<Verifier> findVerifier(String id, String key){
return Lists.newArrayList(hmacVerifier);
}
};
VerifierProviders locators = new VerifierProviders();
locators.setVerifierProvider(SignatureAlgorithm.HS256, hmacLocator);
net.oauth.jsontoken.Checker checker = new net.oauth.jsontoken.Checker(){
#Override
public void check(JsonObject payload) throws SignatureException {
// don't throw - allow anything
}
};
//Ignore Audience does not mean that the Signature is ignored
JsonTokenParser parser = new JsonTokenParser(locators,
checker);
JsonToken jt;
try {
jt = parser.verifyAndDeserialize(token);
} catch (SignatureException e) {
throw new RuntimeException(e);
}
JsonObject payload = jt.getPayloadAsJsonObject();
TokenInfo t = new TokenInfo();
String issuer = payload.getAsJsonPrimitive("iss").getAsString();
String userIdString = payload.getAsJsonObject("info").getAsJsonPrimitive("userId").getAsString();
if (issuer.equals(ISSUER) && !StringUtils.isBlank(userIdString))
{
t.setUserId(new ObjectId(userIdString));
t.setIssued(new DateTime(payload.getAsJsonPrimitive("iat").getAsLong()));
t.setExpires(new DateTime(payload.getAsJsonPrimitive("exp").getAsLong()));
return t;
}
else
{
return null;
}
} catch (InvalidKeyException e1) {
throw new RuntimeException(e1);
}
}
}
public class TokenInfo {
private ObjectId userId;
private DateTime issued;
private DateTime expires;
public ObjectId getUserId() {
return userId;
}
public void setUserId(ObjectId userId) {
this.userId = userId;
}
public DateTime getIssued() {
return issued;
}
public void setIssued(DateTime issued) {
this.issued = issued;
}
public DateTime getExpires() {
return expires;
}
public void setExpires(DateTime expires) {
this.expires = expires;
}
}
This is based on code here: https://developers.google.com/wallet/instant-buy/about-jwts
And Here: https://code.google.com/p/wallet-online-sample-java/source/browse/src/com/google/wallet/online/jwt/util/WalletOnlineService.java?r=08b3333bd7260b20846d7d96d3cf15be8a128dfa
IETF has suggested jose libs on it's wiki:
http://trac.tools.ietf.org/wg/jose/trac/wiki
I would highly recommend using them for signing. I am not a Java guy, but seems like jose4j seems like a good option. Has nice examples as well: https://bitbucket.org/b_c/jose4j/wiki/JWS%20Examples
Update: jwt.io provides a neat comparison of several jwt related
libraries, and their features. A must check!
I would love to hear about what other java devs prefer.
I found this to be small and complete https://github.com/auth0/java-jwt
This page keeps references to implementations in various languages, including Java, and compares features: http://kjur.github.io/jsjws/index_mat.html
If you only need to parse unsigned unencrypted tokens you could use this code:
boolean parseJWT_2() {
String authToken = getToken();
String[] segments = authToken.split("\\.");
String base64String = segments[1];
int requiredLength = (int)(4 * Math.ceil(base64String.length() / 4.0));
int nbrPaddings = requiredLength - base64String.length();
if (nbrPaddings > 0) {
base64String = base64String + "====".substring(0, nbrPaddings);
}
base64String = base64String.replace("-", "+");
base64String = base64String.replace("_", "/");
try {
byte[] data = Base64.decode(base64String, Base64.DEFAULT);
String text;
text = new String(data, "UTF-8");
tokenInfo = new Gson().fromJson(text, TokenInfo.class);
} catch (Exception e) {
e.printStackTrace();
return false;
}
return true;
}
https://github.com/networknt/jsontoken
This is a fork of original google jsontoken
It has not been updated since Sep 11, 2012 and depends on some old packages.
What I have done:
Convert from Joda time to Java 8 time. So it requires Java 8.
Covert Json parser from Gson to Jackson as I don't want to include two Json parsers to my projects.
Remove google collections from dependency list as it is stopped long time ago.
Fix thread safe issue with Java Mac.doFinal call.
All existing unit tests passed along with some newly added test cases.
Here is a sample to generate token and verify the token. For more information, please check https://github.com/networknt/light source code for usage.
I am the author of both jsontoken and Omni-Channel Application Framework.

Convenient way to extract data from the MtGox/PubNub JSON API?

I'm using the PubNub API with Java for pulling data from MtGox.
When retrieving data, the API delivers it in form of a JSONObject, which represents a tree structure of JSON data. Trying to extract bits of information directly from a JSONObject produces ugly code with lots of calls to getJSONObject(String), for which again exceptions might need to be handled.
Therefor, I'm looking for a convenient way to extract information from the JSONObject responses. So far, I've come across the possibility to convert the JSONObject into a POJO and then access the POJO. For conversion, I found the ObjectMapper from the Jackson library, which does a nice job here:
public void successCallback(String channel, Object message) {
JSONObject messageJson = (JSONObject) message;
ObjectMapper mapper = new ObjectMapper();
Message myMessage = mapper.readValue(messageJson.toString(), Message.class);
// do stuff with myMessage here
}
This approach has the disadvantage that I have to write my own POJO classes, e.g. the Message class in the above example, because I could not find these classes ready to use anywhere.
How to conveniently access the information stored in the JSONObject?
PubNub Java Class for MtGox JSON API
It's easy to create a ready made Java Class for ingesting the live feed provided by Mt.Gox This is a work-in-progress post to show you how to access the PubNub Data Feed from Mt.Gox as shown in the Dev Console live feed!
Official Bitcoin Wiki JSON Streaming API
We will be working from the Bitcoin wiki feed instructions provided by Bitcoin official Wiki: https://en.bitcoin.it/wiki/MtGox/API/Pubnub - continue reading below the screenshot to continue.
To see the live real-time data feed we will be using, please checkout the following two links:
Live Feed Trade Events (Buy/Sell Feed): https://www.pubnub.com/console?sub=sub-c-50d56e1e-2fd9-11e3-a041-02ee2ddab7fe&pub=demo&channel=dbf1dee9-4f2e-4a08-8cb7-748919a71b21&origin=pubsub.pubnub.com&ssl=true
Live Feed Ticker Updates (Price Changes): https://www.pubnub.com/console?sub=sub-c-50d56e1e-2fd9-11e3-a041-02ee2ddab7fe&pub=demo&channel=d5f06780-30a8-4a48-a2f8-7ed181b4a13f&origin=pubsub.pubnub.com&ssl=true
Trade Lag Example: https://www.mtgox.com/lag.html
PubNub Java SDK Docs
We will be using the PubNub Java SDK Docs
http://www.pubnub.com/docs/java/javase/overview/data-push.html
Specifically we'll be using the mtgox.subcribe(...) instance method to focus our efforts which looks like the following:
Download JAR or Source: https://github.com/pubnub/mtgox
import org.json.JSONObject;
import com.pubnub.mtgox.MtGox;
import com.pubnub.mtgox.MtGoxCallback;
public class PubnubMtGoxSample {
public static void main(String[] args) {
MtGox mtgx = new MtGox();
mtgx.subscribe("ticker.BTCUSD", new MtGoxCallback(){
#Override
public void callback(JSONObject data) {
try {
String channel_name = data.getString("channel_name");
String avg_value = data.getJSONObject("ticker").getJSONObject("avg").getString("value");
System.out.println(channel_name + " : " + avg_value);
} catch (Exception e) {}
}});
}
}
See Full MtGox Example with Java Source Code - https://github.com/pubnub/mtgox/blob/master/java/examples/PubnubMtGoxSample.java
To compile the example got to https://github.com/pubnub/mtgox/tree/master/java and run
javac -cp Pubnub-MtGox.jar:libs/json-20090211.jar examples/PubnubMtGoxSample.java
And then to RUN:
java -cp .:examples/:Pubnub-MtGox.jar:Pubnub-StandardEdition-3.5.6.jar:libs/json-20090211.jar:libs/bcprov-jdk15on-1.47.jar:libs/slf4j-api-1.7.5.jar:libs/slf4j-nop-1.7.5.jar PubnubMtGoxSample
The Concept
For me, the best solution was to convert the JSONObjects from the PubNub API to bean classes which I found in the MtGox module of the XChange library.
Admitted, this approach adds quite a bit of glue code as you can see at the end of this answer, but I think it's worth the trouble, because after the conversion, the code gets much simpler. E.g. for getting the rate and currency from the ticker at which BTC was last traded, you can simply write
ticker.getLast().getValue()
and
ticker.getLast().getCurrency()
How To Do It
The mtgox module of the XChange library is available as a maven artifact, which is very convenient. You only need to add this module as a dependency to your project and the project setup is done.
In the xchange-mtgox module, you will find the package com.xeiam.xchange.mtgox.v2.dto.marketdata with the two calsses MtGoxTrade and MtGoxTicker.
Converting from JSONObject to one of these classes is easy with the Jackson ObjectMapper. As an advantage, the Jackson library is automatically imported as a transitive dependency of the xchange-mtgox maven artifact. That means that if you're using maven, you don't even have to write a single line of code to add it to your project.
Below is a complete runnable Example. Most is standard code for using PubNub. The important bits are between the marks // BEGIN OF IMPORTANT PART and // END OF IMPORTANT PART.
public class PubNubMtGoxBeanExample {
private static final String MTGOXPUBNUB_SUBSCRIBE_KEY = "sub-c-50d56e1e-2fd9-11e3-a041-02ee2ddab7fe";
private static final String MTGOXPUBNUB_BTCEUR_CHANNEL = "0bb6da8b-f6c6-4ecf-8f0d-a544ad948c15";
private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
static {
OBJECT_MAPPER.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
}
public static void main(String[] args) throws PubnubException {
Pubnub pubnub = new Pubnub("demo", MTGOXPUBNUB_SUBSCRIBE_KEY);
pubnub.subscribe(MTGOXPUBNUB_BTCEUR_CHANNEL, new Callback() {
#Override
public void successCallback(String channel, Object message) {
// BEGIN OF IMPORTANT PART
JSONObject messageJson = (JSONObject) message;
JSONObject tickerJson;
try {
tickerJson = messageJson.getJSONObject("ticker");
} catch (JSONException e) {
throw new RuntimeException(e);
}
MtGoxTicker ticker;
try {
// the following line is the most important, because it maps from the JSONObject to the MtGoxTicker class
ticker = OBJECT_MAPPER.readValue(tickerJson.toString(), MtGoxTicker.class);
} catch (JsonParseException e) {
throw new RuntimeException(e);
} catch (JsonMappingException e) {
throw new RuntimeException(e);
} catch (IOException e) {
throw new RuntimeException(e);
}
String currency = ticker.getLast().getCurrency();
BigDecimal value = ticker.getLast().getValue();
System.out.println(currency + " " + value);
// END OF IMPORTANT PART
}
#Override
public void connectCallback(String channel, Object message) {
System.out.println("connectCallback on channel:" + channel + " : " + message.getClass() + " : " + message.toString());
}
#Override
public void disconnectCallback(String channel, Object message) {
System.out.println("disconnectCallback on channel:" + channel + " : " + message.getClass() + " : " + message.toString());
}
#Override
public void reconnectCallback(String channel, Object message) {
System.out.println("reconnectCallback on channel:" + channel + " : " + message.getClass() + " : " + message.toString());
}
#Override
public void errorCallback(String channel, PubnubError error) {
System.out.println("errorCallback on channel " + channel + " : " + error.toString());
}
});
}
}
For clarity, I've removed the imports, which you can add back in with the appropriate hotkeys in most IDEs (it's Ctrl+Shift+O in Eclipse).
Morevoer, note that there is a performance penalty in this code, which can be avoided by following the answer to the question
How to efficiently map a org.json.JSONObject to a POJO?

GSON: knowing what type of object to convert to?

I'm looking into using Google's GSON for my Android project that will request JSON from my web server. The JSON returned will be either...
1) A successful response of a known type (e.g.: class "User"):
{
"id":1,
"username":"bob",
"created_at":"2011-01-31 22:46:01",
"PhoneNumbers":[
{
"type":"home",
"number":"+1-234-567-8910"
},
{
"type":"mobile",
"number":"+1-098-765-4321"
}
]
}
2.) An unsuccessful response, which will always take on the same basic structure below.
{
"error":{
"type":"Error",
"code":404,
"message":"Not Found"
}
}
I'd like GSON to convert to the correct type depending on the existence of the error key/value pair above. The most practical way I can think to do this is as follows, but I'm curious if there's a better way.
final String response = client.get("http://www.example.com/user.json?id=1");
final Gson gson = new Gson();
try {
final UserEntity user = gson.fromJson(response, UserEntity.class);
// do something with user
} catch (final JsonSyntaxException e) {
try {
final ErrorEntity error = gson.fromJson(response, ErrorEntity.class);
// do something with error
} catch (final JsonSyntaxException e) {
// handle situation where response cannot be parsed
}
}
This is really just pseudocode though, because in the first catch condition, I'm not sure how to test whether the key error exists in the JSON response. So I guess my question is twofold:
Can I / how can I use GSON to test the existence of a key, and make a decision on how to parse based upon that?
Is this what others in a similar situation are doing with GSON, or is there a better way?
What you'd normally want to do is to have your server return an actual error code along with the JSON error response. Then you read the response as an ErrorEntity if you get an error code and as a UserEntity if you get 200. Obviously this requires a little more dealing with the details of communication with the server than just turning a URL in to a String, but that's how it is.
That said, I believe another option would be to use a custom JsonDeserializer and a class that can return either a value or an error.
public class ValueOrErrorDeserializer<V> implements JsonDeserializer<ValueOrError<V>> {
public ValueOrError<V> deserialize(JsonElement json, Type typeOfT,
JsonDeserializationContext context) {
JsonObject object = json.getAsJsonObject();
JsonElement error = object.get("error");
if (error != null) {
ErrorEntity entity = context.deserialize(error, ErrorEntity.class);
return ValueOrError.<V>error(entity);
} else {
Type valueType = ((ParameterizedType) typeOfT).getActualTypeArguments()[0];
V value = (V) context.deserialize(json, valueType);
return ValueOrError.value(value);
}
}
}
You'd then be able to do something like this:
String response = ...
ValueOrError<UserEntity> valueOrError = gson.fromJson(response,
new TypeToken<ValueOrError<UserEntity>>(){}.getType());
if (valueOrError.isError()) {
ErrorEntity error = valueOrError.getError();
...
} else {
UserEntity user = valueOrError.getValue();
...
}
I haven't tried that code out, and I'd still recommend using the HTTP error code, but it gives you an example of how you can use a JsonDeserializer to decide what to do with some JSON.

Categories