I am constructing a POC. And I created a pass through proxy service for Google Plus.
Without using any proxy service I get this is my output :
{
"kind":"plus#person",
"etag":"\"ExituU7aUpmkkfyD52VulzptThw/4J1clegrhxYC2fsJOu2XWCs1Ewg\"",
"id":"117488614303967062311",
"displayName":"Abhi NeoN",
"name":{
"familyName":"NeoN",
"givenName":"Abhi"
},
"tagline":"hey guys ! ssup!! check out ma recnt videos... uploaded",
"gender":"male",
"aboutMe":"\u003cb\u003ehie, abhishek - ma full name \u003c/b\u003e\u003cdiv\u003e\u003cb\u003em a DANCER ,\u003c/b\u003e\u003c/div\u003e\u003cdiv\u003e\u003cb\u003ei luv ma dancing .\u003c/b\u003e\u003c/div\u003e\u003cdiv\u003e\u003cb\u003ei care ma dancing ,\u003c/b\u003e\u003c/div\u003e\u003cdiv\u003e\u003cb\u003ei jus hv a gr8 thng in me dats ma dancing.\u003c/b\u003e\u003c/div\u003e",
"relationshipStatus":"single",
"url":"https://plus.google.com/117488614303967062311",
"image":{
"url":"https://lh6.googleusercontent.com/-tF-ip0tUxD4/AAAAAAAAAAI/AAAAAAAAAAA/WKI3USUh_DA/photo.jpg?sz=50"
},
"urls":[
{
"value":"https://plus.google.com/117488614303967062311",
"type":"profile"
},
{
"value":"https://www.googleapis.com/plus/v1/people/117488614303967062311",
"type":"json"
}
],
"organizations":[
{
"name":"our lady of nazareth high school",
"title":"science",
"type":"school"
},
{
"name":"",
"title":"BLUEBYTES",
"type":"work"
}
]
}
But when I try to do the same using a simple pass through service I get only :
{
"kind":"plus#person"
}
I read on the wso2esb site that they had a bug and the explanation given to resolve the bug was that json data received was not in the proper format.
But now how do I resolve the problem. I mean is their any way I can manipulate the json data before the esb converts it into json data.
We have solved this issue in the latest release of ESB (version 4.5.0). By default it comes with JSONMessageFormatter/JSONBuilder that can handle JSON payloads with multiple keys.
We also came up with another solution for handling message flows that involve different types of JSON <--> XML (or JSON <--> JSON) conversions. JSONStreamBuilder and JSONStreamFormatter can be used to implement such scenarios with the 'script' mediator. Have a look at sample #441 in ESB 4.5.0.
To run sample #441;
Add JSONStreamBuilder and JSONStreamFormatter as the builder and
formatter for JSON in repository/conf/axis2/axis2.xml file
Deploy SimpleStockQuoteService
Start the sample axis2server
Run the JSON client with 'ant newjsonclient'
This is one of the limitations of the current axis2 JSON builder/formatter. We are currently working on a new builder/formatter pair for JSON that does not convert JSON <-> XML. Instead it(builder) stores the JSON message as a stream and the script mediator can be used to build a JSON object out of that stream. For example, if we send {"a" : "x", "b" : "y"} as the request, within the ESB, we can manipulate this request as a JSON object with javascript.
var a = mc.getJSON().a.toString();
var b = mc.getJSON().b.toString();
mc.setPayloadXML(
<m:A xmlns:m="http://example.json">
<m:a>{a}</m:a>
<m:b>{b}</m:b>
</m:A>);
Similarly mc.setJSON() method can be used to set arbitrary JSON objects.
The only way to reliably convert json to xml and back again is via the use of type hints in the xml. the default converter does not do this. it
1. drops everything after the first property
2. confuses single element lists with properties when going from xml to json
i have reimplemented the transconversion classes using the json-util library, which converts the json to xml containing type hints as element attributes, to ensure no ambiguity.
in this way we can smart proxy (ie content route and mediate on transport and payload) for ALL json based rest services through WSO2 with no issues
This solves the problem (I think camel does it this way by default).
Here is the pom file and code:
place the jar into /repository/components/lib
you must update the messageformatter and messagebuilder mappings for content type "application/json" in axis2.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<name>wso2 json/xml converter</name>
<groupId>x.y.z</groupId>
<artifactId>wso2converter</artifactId>
<version>1.0.0-SNAPSHOT</version>
<packaging>jar</packaging>
<properties>
<jdk.version>1.6</jdk.version>
</properties>
<build>
<finalName>wso2converter</finalName>
<resources>
<resource>
<filtering>false</filtering>
<directory>src/main/resources</directory>
</resource>
</resources>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>${jdk.version}</source>
<target>${jdk.version}</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<version>1.0.1</version>
<executions>
<execution>
<id>enforce-jdk</id>
<phase>validate</phase>
<goals>
<goal>display-info</goal>
<goal>enforce</goal>
</goals>
<configuration>
<rules>
<requireJavaVersion>
<version>[${jdk.version},)</version>
</requireJavaVersion>
</rules>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>net.sf.json-lib</groupId>
<artifactId>json-lib</artifactId>
<version>2.3</version>
<classifier>jdk15</classifier>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-io</artifactId>
<version>1.3.2</version>
<type>jar</type>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.ws.commons.axiom</groupId>
<artifactId>axiom-api</artifactId>
<version>1.2.13</version>
</dependency>
<dependency>
<groupId>org.apache.axis2</groupId>
<artifactId>axis2-kernel</artifactId>
<version>1.6.2</version>
</dependency>
<dependency>
<groupId>xom</groupId>
<artifactId>xom</artifactId>
<version>1.1</version>
</dependency>
<dependency>
<groupId>org.apache.synapse</groupId>
<artifactId>synapse-core</artifactId>
<version>2.1.0</version>
</dependency>
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-json</artifactId>
<version>1.1.5</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.8.2</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.13</version>
<!--scope>provided</scope-->
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.4</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
package a.b.wso2;
import java.io.InputStream;
import net.sf.json.JSON;
import net.sf.json.JSONSerializer;
import net.sf.json.xml.XMLSerializer;
import org.apache.axiom.om.OMElement;
import org.apache.axiom.om.util.AXIOMUtil;
import org.apache.axis2.AxisFault;
import org.apache.axis2.builder.Builder;
import org.apache.axis2.context.MessageContext;
import org.apache.commons.io.IOUtils;
import org.apache.log4j.BasicConfigurator;
import org.apache.log4j.Logger;
public class WsoJtoX implements Builder {
Logger logger = Logger.getLogger("a.b.wso2converter");
#Override
public OMElement processDocument(InputStream is, String contentType,
MessageContext messageContext) throws AxisFault {
String jsonData = "";
try {
jsonData = IOUtils.toString(is,"UTF-8");
String output = process(jsonData);
OMElement e = AXIOMUtil.stringToOM(output);
return e;
} catch (Exception e) {
logger.error("error converting json string " + jsonData, e);
if (e instanceof AxisFault) {
throw (AxisFault) e;
}
throw new AxisFault("(B"+counter+") error converting json to xml", e);
}
}
static int counter=0;
public String process(String jsonData) throws AxisFault {
try {
String tran = "__ns__";
jsonData=jsonData.replace("\r", "").trim();
//jsonData=jsonData.replace("\n", "");
String decoded = (jsonData.replaceAll("\"([a-zA-Z0-9_]*)\\:([a-zA-Z0-9]*)\"(\\s*)(:)", "\"$1" + tran + "$2\"$3:"));
counter++;
if (logger.isDebugEnabled()) {
logger.debug("\n>>>>> (B"+counter+") converting json\n " + jsonData + "\n====");
}
XMLSerializer serializer = new XMLSerializer();
JSON json = JSONSerializer.toJSON(decoded);
String xml = serializer.write(json);
//add in the soap stuff
StringBuilder sb = new StringBuilder();
sb.append("<soap:Envelope xmlns:soap=\"http://www.w3.org/2001/12/soap-envelope\" soap:encodingStyle=\"http://www.w3.org/2001/12/soap-encoding\"> <soap:Body>");
sb.append(xml.replace("<?xml version=\"1.0\" encoding=\"UTF-8\"?>", ""));
sb.append("</soap:Body></soap:Envelope>");
if (logger.isDebugEnabled()) {
logger.debug("\n==== (B"+counter+") to xml\n" + sb.toString()+"\n<<<<<");
}
return sb.toString();
} catch (Exception e) {
throw new AxisFault("(B"+counter+") error transforming json to xml", e);
}
}
}
package a.b.wso2;
import java.io.IOException;
import java.io.OutputStream;
import java.io.UnsupportedEncodingException;
import java.net.URL;
import net.sf.json.JSON;
import net.sf.json.xml.XMLSerializer;
import org.apache.axiom.om.OMElement;
import org.apache.axiom.om.OMOutputFormat;
import org.apache.axiom.om.util.AXIOMUtil;
import org.apache.axis2.AxisFault;
import org.apache.axis2.Constants;
import org.apache.axis2.context.MessageContext;
import org.apache.axis2.transport.MessageFormatter;
import org.apache.log4j.BasicConfigurator;
import org.apache.log4j.Logger;
public class WsoXtoJ implements MessageFormatter {
Logger logger = Logger.getLogger("a.b.wso2converter");
private static int counter=0;
public String convert(String xData) {
counter++;
if (logger.isDebugEnabled()) {
logger.debug("\n]]]]] (A"+counter+") converting xml\n " + xData + "\n-----");
}
try {
String tran = "__ns__";
XMLSerializer serializer = new XMLSerializer();
OMElement e = AXIOMUtil.stringToOM(xData);
OMElement b = (OMElement) e.getChildrenWithLocalName("Body").next();
b = (OMElement) b.getChildElements().next();
String xfrag = b.toStringWithConsume();
String str = "";
JSON j = serializer.read(xfrag);
str = j.toString();
String nstr = str.replaceAll("\"([a-zA-Z0-9_]+)" + tran + "([a-zA-Z0-9]+)\"(\\s*)(:)", "\"$1:$2\"$3:"); //", "\"$1:$2\"");
if (logger.isDebugEnabled()) {
logger.debug("\n----- (A"+counter+") to json\n" + nstr+"\n[[[[[");
}
return nstr;
} catch (Exception e) {
throw new RuntimeException(e);
}
}
#Override
public String formatSOAPAction(MessageContext msgCtxt, OMOutputFormat format,
String soapActionString) {
return null;
}
#Override
public byte[] getBytes(MessageContext ctx, OMOutputFormat format)
throws AxisFault {
String env="";
try {
OMElement element = ctx.getEnvelope().getBody().getFirstElement();
String payload = this.convert(element.toString());
return payload.getBytes(format.getCharSetEncoding());
} catch (UnsupportedEncodingException e) {
logger.error("(A"+counter+") error converting xml to json "+ctx.getEnvelope().toString());
throw AxisFault.makeFault(e);
}
}
#Override
public String getContentType(MessageContext msgCtxt, OMOutputFormat format,
String soapActionString) {
String contentType = (String) msgCtxt.getProperty(Constants.Configuration.CONTENT_TYPE);
String encoding = format.getCharSetEncoding();
if (contentType == null) {
contentType = (String) msgCtxt.getProperty(Constants.Configuration.MESSAGE_TYPE);
}
if (encoding != null) {
contentType += "; charset=" + encoding;
}
return contentType;
}
#Override
public URL getTargetAddress(MessageContext msgCtxt, OMOutputFormat format,
URL targetURL) throws AxisFault {
return targetURL;
}
#Override
public void writeTo(MessageContext msgCtxt, OMOutputFormat format,
OutputStream out, boolean preserve) throws AxisFault {
try {
out.write(this.getBytes(msgCtxt, format));
out.flush();
} catch (IOException e) {
throw AxisFault.makeFault(e);
}
}
}
I had the same problem.
In my experience, the JSON parser for WSO2 ESB (based on Axis2-json) supports only a subset of JSON:
The JSON has to start with "{", i.e. there can't be a JSONArray at the root.
Only the first key-value pair will be considered. This is because JSON is mapped to XML-like datastructure, and XML must have a root, so the first key-value pair is considered as root.
The value of the first key-value pair must not be an array. This because the converter has to know which XML tag should be used for each value:
e.g.: ... { "key": ["val1", "val2", ...]} -> <key>val1</key><key>val2</key>....
I have the same problem here and want to find a fix for this. My thoughts are to create a new JSONBuilder (the parser which builds the internal SOAP message construct) and the JSONFormatter (the serializer) to use a virtual root (e.g. { "root" : ... } ) to fake the parser.
Related
I am trying to execute a simple Java example that uses the AWS Polly service. I am using the code provided by AWS on their documentation. I created a simple Maven Project using the following -**
1. group id - com.amazonaws.polly
2. artifact id - java-demo
3. version - 0.0.1-SNAPSHOT
Following is my project structure -
Following is my pom.xml -
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.amazonaws.polly</groupId>
<artifactId>java-demo</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependencies>
<!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-polly -->
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-polly</artifactId>
<version>1.11.77</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.googlecode.soundlibs/jlayer -->
<dependency>
<groupId>com.googlecode.soundlibs</groupId>
<artifactId>jlayer</artifactId>
<version>1.0.1-1</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.2.1</version>
<executions>
<execution>
<goals>
<goal>java</goal>
</goals>
</execution>
</executions>
<configuration>
<mainClass>com.amazonaws.demos.polly.PollyDemo</mainClass>
</configuration>
</plugin>
</plugins>
</build>
</project>
Following is my java class -
package com.amazonaws.demos.polly;
import java.io.IOException;
import java.io.InputStream;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
import com.amazonaws.regions.Region;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.polly.AmazonPollyClient;
import com.amazonaws.services.polly.model.DescribeVoicesRequest;
import com.amazonaws.services.polly.model.DescribeVoicesResult;
import com.amazonaws.services.polly.model.OutputFormat;
import com.amazonaws.services.polly.model.SynthesizeSpeechRequest;
import com.amazonaws.services.polly.model.SynthesizeSpeechResult;
import com.amazonaws.services.polly.model.Voice;
import javazoom.jl.player.advanced.AdvancedPlayer;
import javazoom.jl.player.advanced.PlaybackEvent;
import javazoom.jl.player.advanced.PlaybackListener;
public class PollyDemo {
private final AmazonPollyClient polly;
private final Voice voice;
private static final String SAMPLE = "Congratulations. You have successfully built this working demo "+
"of Amazon Polly in Java. Have fun building voice enabled apps with Amazon Polly (that's me!), and always"+
"look at the AWS website for tips and tricks on using Amazon Polly and other great services from AWS";
public PollyDemo(Region region) {
//Didn't work
//AWSCredentials credentials = new BasicAWSCredentials("someAccessKey","someSecretKey");
//polly = new AmazonPollyClient(credentials);
//Didn't work
// create an Amazon Polly client in a specific region
polly = new AmazonPollyClient(new DefaultAWSCredentialsProviderChain(),
new ClientConfiguration());
polly.setRegion(region);
// Create describe voices request.
DescribeVoicesRequest describeVoicesRequest = new DescribeVoicesRequest();
// Synchronously ask Amazon Polly to describe available TTS voices.
DescribeVoicesResult describeVoicesResult = polly.describeVoices(describeVoicesRequest);
voice = describeVoicesResult.getVoices().get(0);
}
public InputStream synthesize(String text, OutputFormat format) throws IOException {
SynthesizeSpeechRequest synthReq =
new SynthesizeSpeechRequest().withText(text).withVoiceId(voice.getId())
.withOutputFormat(format);
SynthesizeSpeechResult synthRes = polly.synthesizeSpeech(synthReq);
return synthRes.getAudioStream();
}
public static void main(String args[]) throws Exception {
//create the test class
PollyDemo helloWorld = new PollyDemo(Region.getRegion(Regions.US_EAST_1));
//get the audio stream
InputStream speechStream = helloWorld.synthesize(SAMPLE, OutputFormat.Mp3);
//create an MP3 player
AdvancedPlayer player = new AdvancedPlayer(speechStream,
javazoom.jl.player.FactoryRegistry.systemRegistry().createAudioDevice());
player.setPlayBackListener(new PlaybackListener() {
#Override
public void playbackStarted(PlaybackEvent evt) {
System.out.println("Playback started");
System.out.println(SAMPLE);
}
#Override
public void playbackFinished(PlaybackEvent evt) {
System.out.println("Playback finished");
}
});
// play it!
player.play();
}
}
I am running the code locally, therefore I have my AWS IAM credentials configured in my system,
My IAM user also has access to AWS Polly service.
I am getting the following error when I run the code -
Exception in thread "main" com.amazonaws.services.polly.model.AmazonPollyException: The security token included in the request is invalid. (Service: AmazonPolly; Status Code: 403; Error Code: UnrecognizedClientException; Request ID: 4d4b01fb-8015-11e8-8e18-4548f95fba92)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1586)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1254)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1035)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:747)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:721)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:704)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:672)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:654)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:518)
at com.amazonaws.services.polly.AmazonPollyClient.doInvoke(AmazonPollyClient.java:668)
at com.amazonaws.services.polly.AmazonPollyClient.invoke(AmazonPollyClient.java:644)
at com.amazonaws.services.polly.AmazonPollyClient.describeVoices(AmazonPollyClient.java:383)
at com.amazonaws.demos.polly.PollyDemo.<init>(PollyDemo.java:39)
at com.amazonaws.demos.polly.PollyDemo.main(PollyDemo.java:54)
I am referring the following AWS Doc for the Polly java example-
https://docs.aws.amazon.com/polly/latest/dg/examples-java.html
Can someone help fix my code? What do I change in my code?
It's a 403 error. Where are you passing the AWS access and secret key? You can try this
* Constructs a new client to invoke service methods on AmazonPolly. A
* credentials provider chain will be used that searches for credentials in
* this order:
* <ul>
* <li>Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY</li>
* <li>Java System Properties - aws.accessKeyId and aws.secretKey</li>
* <li>Instance profile credentials delivered through the Amazon EC2
* metadata service</li>
* </ul>
* <p>
* All service calls made using this new client object are blocking, and
* will not return until the service call completes.
*
* #see DefaultAWSCredentialsProviderChain
*/
public AmazonPollyClient() {
this(new DefaultAWSCredentialsProviderChain(), new ClientConfiguration());
}
https://github.com/aws/aws-sdk-android/blob/master/aws-android-sdk-polly/src/main/java/com/amazonaws/services/polly/AmazonPollyClient.java
Replace
new DefaultAWSCredentialsProviderChain()
with
AWSStaticCredentialsProvider(new BasicAWSCredentials("AccessKey", "Secret Key"))
I noticed there is no POM file in the Github repository for this service. We will fix that. Here is a POM file that works.
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>pollyJ1Project</groupId>
<artifactId>pollyJ1Project</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<java.version>1.8</java.version>
</properties>
<dependencies>
<!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-polly -->
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-polly</artifactId>
<version>1.11.774</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.awaitility/awaitility -->
<!-- https://mvnrepository.com/artifact/org.awaitility/awaitility -->
<dependency>
<groupId>org.awaitility</groupId>
<artifactId>awaitility</artifactId>
<version>4.0.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.awaitility</groupId>
<artifactId>awaitility</artifactId>
<version>4.0.2</version>
<scope>compile</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build>
Also - here is an example you can try:
package com.amazonaws.polly.samples;
import com.amazonaws.services.polly.AmazonPolly;
import com.amazonaws.services.polly.AmazonPollyClientBuilder;
import com.amazonaws.services.polly.model.DescribeVoicesRequest;
import com.amazonaws.services.polly.model.DescribeVoicesResult;
public class DescribeVoicesSample {
public static void main(String[] args) {
AmazonPolly client = AmazonPollyClientBuilder.defaultClient();
describeVoices(client);
}
public static void describeVoices(AmazonPolly client ) {
DescribeVoicesRequest allVoicesRequest = new DescribeVoicesRequest();
DescribeVoicesRequest enUsVoicesRequest = new DescribeVoicesRequest()
.withLanguageCode("en-US");
try {
String nextToken;
do {
DescribeVoicesResult allVoicesResult =
client.describeVoices(allVoicesRequest);
nextToken = allVoicesResult.getNextToken();
allVoicesRequest.setNextToken(nextToken);
System.out.println("All voices: " + allVoicesResult.getVoices());
} while (nextToken != null);
do {
DescribeVoicesResult enUsVoicesResult = client.describeVoices(enUsVoicesRequest);
nextToken = enUsVoicesResult.getNextToken();
enUsVoicesRequest.setNextToken(nextToken);
System.out.println("en-US voices: " + enUsVoicesResult.getVoices());
} while (nextToken != null);
} catch (Exception e) {
System.err.println("Exception caught: " + e);
}
}
}
The above code is V1. You can find V2 code examples here:
https://github.com/awsdocs/aws-doc-sdk-examples/tree/master/javav2/example_code/polly
I found a nice working demo here https://youtu.be/WMMSQAn_vHI. It is working java example for AWS Polly service using new DefaultAWSCredentialsProviderChain() only.
I tried to create a simple class to create a table and add some columns with HBase and Google app engine.
I have already created a project and an instance in Google Cloud platform.
I cloned this repository : https://github.com/GoogleCloudPlatform/cloud-bigtable-examples/blob/master/java/hello-world/src/main/java/com/example/cloud/bigtable/helloworld/HelloWorld.java
It works like a charm to create a table in my instance.
But when I'm trying to create a new maven project with the same config, it doesn't work, I can't create anything.
I got this issue :
InvocationTargetException: Could not find an appropriate constructor for com.google.cloud.bigtable.hbase1_x.BigtableConnection: com.google.common.util.concurrent.MoreExecutors.platformThreadFactory()Ljava/util/concurrent/ThreadFactory;
here is my AudioBridgeData.java file :
package com.xxx.xxx;
import com.google.cloud.bigtable.hbase.BigtableConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.util.Bytes;
import java.io.IOException;
/**
* A minimal application that connects to Cloud Bigtable using the native HBase API
* and performs some basic operations.
*/
public class AudioBridgeData {
private static final byte[] TABLE_NAME = Bytes.toBytes("audio-bridge");
private static final byte[] COLUMN_FAMILY_NAME = Bytes.toBytes("phone-number");
private static final byte[] COLUMN_NAME = Bytes.toBytes("number");
private static final String[] NUMBERS =
{ "+33697584976", "+19553560976", "+4879665676" };
/**
* Connects to Cloud Bigtable, runs some basic operations and prints the results.
*/
private static void doAudioBridge(String projectId, String instanceId) {
try (Connection connection = BigtableConfiguration.connect(projectId, instanceId)) {
Admin admin = connection.getAdmin();
HTableDescriptor descriptor = new HTableDescriptor(TableName.valueOf(TABLE_NAME));
descriptor.addFamily(new HColumnDescriptor(COLUMN_FAMILY_NAME));
print("Create table " + descriptor.getNameAsString());
admin.createTable(descriptor);
Table table = connection.getTable(TableName.valueOf(TABLE_NAME));
print("Write some numbers to the table");
for (int i = 0; i < NUMBERS.length; i++) {
Put put = new Put(Bytes.toBytes(i));
put.addColumn(COLUMN_FAMILY_NAME, COLUMN_NAME, Bytes.toBytes(NUMBERS[i]));
table.put(put);
}
int rowKey = 0;
Result getResult = table.get(new Get(Bytes.toBytes(rowKey)));
String number = Bytes.toString(getResult.getValue(COLUMN_FAMILY_NAME, COLUMN_NAME));
System.out.println("Get a single number by row key");
System.out.printf("\t%s = %s\n", rowKey, number);
Scan scan = new Scan();
print("Scan for all numbers:");
ResultScanner scanner = table.getScanner(scan);
for (Result row : scanner) {
byte[] valueBytes = row.getValue(COLUMN_FAMILY_NAME, COLUMN_NAME);
System.out.println('\t' + Bytes.toString(valueBytes));
}
} catch (IOException e) {
System.err.println("Exception while running HelloWorld: " + e.getMessage());
e.printStackTrace();
System.exit(1);
}
System.exit(0);
}
private static void print(String msg) {
System.out.println("Number: " + msg);
}
public static void main(String[] args) {
String projectId = requiredProperty("bigtable.projectID");
String instanceId = requiredProperty("bigtable.instanceID");
doAudioBridge(projectId, instanceId);
}
private static String requiredProperty(String prop) {
String value = System.getProperty(prop);
if (value == null) {
throw new IllegalArgumentException("Missing required system property: " + prop);
}
return value;
}
}
Here is my pom.xml file :
<?xml version="1.0" encoding="UTF-8"?>
<project>
<modelVersion>4.0.0</modelVersion>
<groupId>com.xxx</groupId>
<artifactId>xxx</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<url>http://maven.apache.org</url>
<name>xxx</name>
<properties>
<bigtable.version>1.0.0-pre1</bigtable.version>
<hbase.version>1.1.5</hbase.version>
<maven.compiler.target>1.8</maven.compiler.target>
<maven.compiler.source>1.8</maven.compiler.source>
</properties>
<repositories>
<repository>
<id>snapshots-repo</id>
<url>https://oss.sonatype.org/content/repositories/snapshots</url>
<releases><enabled>false</enabled></releases>
<snapshots><enabled>true</enabled></snapshots>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>com.google.cloud.bigtable</groupId>
<artifactId>bigtable-hbase-1.x</artifactId>
<version>${bigtable.version}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<version>1.1.33.Fork26</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>0.98.11-hadoop2</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.5.0</version>
<configuration>
<mainClass>com.xxx.xxx.AudioBridgeData</mainClass>
</configuration>
</plugin>
</plugins>
</build>
</project>
And I tried to run this command :
mvn exec:java -Dbigtable.projectID=xxx -Dbigtable.instanceID=quickstart-instance
Thanks a lot for your help ! :)
I had the same error message, weirdly it was caused by missing application credentials. If its the same issue you should set this environment variable:
GOOGLE_APPLICATION_CREDENTIALS
It should be set to the location of the client credentials file you (might have) downloaded after creating a service account key. This is a good page:
https://developers.google.com/identity/protocols/application-default-credentials
My requirement is to convert a plain Java object to JSON format -
The Java object has following format -
Record Id: 168349200
Name: jane
City: abababa
State: ababab
Insertion Date: 18/04/2017 10:16:17
I have gone through some tutorials and found that GSON library is a way to do it. I tried to install the gson jar in my Eclipse project (using Eclipse Mars Release (4.5.0)).
But when I do -
import com.google.gson.Gson
in the class where I want to do the conversion of the Java object to JSON it throws an exception.
I think I have not added the GSON jar file properly.
Can some one please help.
Thanks.
Try:
Gson gson = new Gson();
String jsonString = null;
try {
jsonString = gson.toJson(javaObject);
} catch (Exception e) {
e.printStackTrace();
}
From Java Object to Json using ObjectMapper:
ObjectMapper objMapper = new ObjectMapper().setSerializationInclusion(Include.NON_NULL);
MyObject mObject = new MyObject();
String jsonObject ="";
mObject.setField1(value1);
mObject.setField2(value2);
mObject.setField3(value3);
try {
jsonObject = objMapper.writeValueAsString(mObject);
System.out.println(jsonObject); //You can then store it in a file
} catch (JsonProcessingException e) {
e.printStackTrace();
}
You should add these imports:
import com.fasterxml.jackson.annotation.JsonInclude.Include;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
And in the pom.xml these lines:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.7.5</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.7.5</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.7.5</version>
</dependency>
I have this JSON-Content:
{"color":null}
And I want to make these Java-Objects out of it (and vice-versa):
Container
|- List<Entry> entries
|- Entry
|- String key = "color"
|- String value = null
My current solution always deserializes "color":null to an empty String. I found other solutions, that will instead deserialize null or empty String to null.
How can I get MOXy (or any other jaxb implementation) to deserialize null as null and empty Strings to empty Strings?
I am using this code:
import java.io.ByteArrayOutputStream;
import java.util.*;
import javax.xml.bind.*;
import javax.xml.bind.annotation.*;
import org.eclipse.persistence.internal.oxm.ByteArraySource;
import org.eclipse.persistence.jaxb.JAXBContextProperties;
import org.eclipse.persistence.oxm.annotations.*;
import org.junit.*;
class Container {
#XmlVariableNode("key")
List<Entry> entries = new ArrayList<Entry>();
}
class Entry {
#XmlTransient
public String key;
#XmlValue
#XmlNullPolicy(nullRepresentationForXml=XmlMarshalNullRepresentation.XSI_NIL, xsiNilRepresentsNull=false)
public String value;
}
public class D {
/** THIS TEST FAILS!!! */
#Test
public void unmarshallNull() throws Exception {
Assert.assertEquals(null, unmarshall("xml", "<root><color xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:nil=\"true\"/></root>"));
Assert.assertEquals(null, unmarshall("json", "{\"color\":null}"));
}
/* All other tests are passing. */
#Test
public void unmarshallEmpty() throws Exception {
Assert.assertEquals("", unmarshall("xml", "<root><color></color></root>"));
Assert.assertEquals("", unmarshall("json", "{\"color\":\"\"}"));
}
#Test
public void unmarshallValue() throws Exception {
Assert.assertEquals("red", unmarshall("xml", "<root><color>red</color></root>"));
Assert.assertEquals("red", unmarshall("json", "{\"color\":\"red\"}"));
}
#Test
public void marshallNull() throws Exception {
Assert.assertEquals("<color xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:nil=\"true\"/>", marshall("xml", null));
Assert.assertEquals("{\"color\":null}", marshall("json", null));
}
#Test
public void marshallEmpty() throws Exception {
Assert.assertEquals("<color></color>", marshall("xml", ""));
Assert.assertEquals("{\"color\":\"\"}", marshall("json", ""));
}
#Test
public void marshallValue() throws Exception {
Assert.assertEquals("<color>red</color>", marshall("xml", "red"));
Assert.assertEquals("{\"color\":\"red\"}", marshall("json", "red"));
}
private static String marshall(String format, String value) throws JAXBException {
// prepare
JAXBContext jc = JAXBContext.newInstance(Container.class);
Marshaller marshaller = jc.createMarshaller();
marshaller.setProperty(JAXBContextProperties.MEDIA_TYPE, "application/"+format);
marshaller.setProperty(Marshaller.JAXB_FRAGMENT, Boolean.TRUE);
marshaller.setProperty(JAXBContextProperties.JSON_INCLUDE_ROOT, false);
// define example data
Container detail = new Container();
Entry entry = new Entry();
entry.key = "color";
entry.value = value;
detail.entries.add(entry);
// marshall
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
marshaller.marshal(detail, outputStream);
return outputStream.toString();
}
private static String unmarshall(String format, String raw) throws JAXBException {
// prepare
JAXBContext jc = JAXBContext.newInstance(Container.class);
Unmarshaller unmarshaller = jc.createUnmarshaller();
unmarshaller.setProperty(JAXBContextProperties.MEDIA_TYPE, "application/"+format);
unmarshaller.setProperty(JAXBContextProperties.JSON_INCLUDE_ROOT, false);
// unmarshall
Container container = unmarshaller.unmarshal(new ByteArraySource(raw.getBytes()), Container.class).getValue();
return container.entries.get(0).value;
}
}
It works for XML but fails for JSON:
Test failure: unmarshallNull
java.lang.AssertionError: expected:<null> but was:<>
I also configured MOXy as my jaxb-provider for the relevant package (jaxb.properties):
javax.xml.bind.context.factory=org.eclipse.persistence.jaxb.JAXBContextFactory
I'm using MOXy 2.22.1 and Java8 (but same behaviour with 2.18, 2.19, 2.20, 2.21, 2.22). My Maven pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>x</groupId>
<artifactId>x</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.glassfish.jersey.media</groupId>
<artifactId>jersey-media-moxy</artifactId>
<version>2.22.1</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
There are things you might want to try.
First, try using #XmlElement(nillable=true) instead of XmlNullPolicy annotation or setting the emptyNodeRepresentsNull parameter.
Second, you might want to write your own XmlAdapter which will do pretty much the same thing - unmarshall empty string to null. The main difference is that you can also manually configure which fields will be unmarshalled with your adapter and which will not, therefore retaining the current behaviour. Please refer to this answer by Blaise Doughan to see how do you wire a custom XmlAdapter to your configuration.
Third, as far as I know Jackson doesn't have this problem. See this answer ot this wiki page about how to set up a custom deserializer for a field in case it does.
I want to POStag an English sentence and do some processing. I would like to use openNLP. I have it installed
When I execute the command
I:\Workshop\Programming\nlp\opennlp-tools-1.5.0-bin\opennlp-tools-1.5.0>java -jar opennlp-tools-1.5.0.jar POSTagger models\en-pos-maxent.bin < Text.txt
It gives output POSTagging the input in Text.txt
Loading POS Tagger model ... done (4.009s)
My_PRP$ name_NN is_VBZ Shabab_NNP i_FW am_VBP 22_CD years_NNS old._.
Average: 66.7 sent/s
Total: 1 sent
Runtime: 0.015s
I hope it installed properly?
Now how do i do this POStagging from inside a java application? I have added the openNLPtools, jwnl, maxent jar to the project but how do i invoke the POStagging?
Here's some (old) sample code I threw together, with modernized code to follow:
package opennlp;
import opennlp.tools.cmdline.PerformanceMonitor;
import opennlp.tools.cmdline.postag.POSModelLoader;
import opennlp.tools.postag.POSModel;
import opennlp.tools.postag.POSSample;
import opennlp.tools.postag.POSTaggerME;
import opennlp.tools.tokenize.WhitespaceTokenizer;
import opennlp.tools.util.ObjectStream;
import opennlp.tools.util.PlainTextByLineStream;
import java.io.File;
import java.io.IOException;
import java.io.StringReader;
public class OpenNlpTest {
public static void main(String[] args) throws IOException {
POSModel model = new POSModelLoader().load(new File("en-pos-maxent.bin"));
PerformanceMonitor perfMon = new PerformanceMonitor(System.err, "sent");
POSTaggerME tagger = new POSTaggerME(model);
String input = "Can anyone help me dig through OpenNLP's horrible documentation?";
ObjectStream<String> lineStream =
new PlainTextByLineStream(new StringReader(input));
perfMon.start();
String line;
while ((line = lineStream.read()) != null) {
String whitespaceTokenizerLine[] = WhitespaceTokenizer.INSTANCE.tokenize(line);
String[] tags = tagger.tag(whitespaceTokenizerLine);
POSSample sample = new POSSample(whitespaceTokenizerLine, tags);
System.out.println(sample.toString());
perfMon.incrementCounter();
}
perfMon.stopAndPrintFinalResult();
}
}
The output is:
Loading POS Tagger model ... done (2.045s)
Can_MD anyone_NN help_VB me_PRP dig_VB through_IN OpenNLP's_NNP horrible_JJ documentation?_NN
Average: 76.9 sent/s
Total: 1 sent
Runtime: 0.013s
This is basically working from the POSTaggerTool class included as part of OpenNLP. The sample.getTags() is a String array that has the tag types themselves.
This requires direct file access to the training data, which is really, really lame.
An updated codebase for this is a little different (and probably more useful.)
First, a Maven POM:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.javachannel</groupId>
<artifactId>opennlp-example</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.apache.opennlp</groupId>
<artifactId>opennlp-tools</artifactId>
<version>1.6.0</version>
</dependency>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>[6.8.21,)</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
And here's the code, written as a test, therefore located in ./src/test/java/org/javachannel/opennlp/example:
package org.javachannel.opennlp.example;
import opennlp.tools.cmdline.PerformanceMonitor;
import opennlp.tools.postag.POSModel;
import opennlp.tools.postag.POSSample;
import opennlp.tools.postag.POSTaggerME;
import opennlp.tools.tokenize.WhitespaceTokenizer;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.net.URL;
import java.nio.channels.Channels;
import java.nio.channels.ReadableByteChannel;
import java.util.stream.Stream;
public class POSTest {
private void download(String url, File destination) throws IOException {
URL website = new URL(url);
ReadableByteChannel rbc = Channels.newChannel(website.openStream());
FileOutputStream fos = new FileOutputStream(destination);
fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
}
#DataProvider
Object[][] getCorpusData() {
return new Object[][][]{{{
"Can anyone help me dig through OpenNLP's horrible documentation?"
}}};
}
#Test(dataProvider = "getCorpusData")
public void showPOS(Object[] input) throws IOException {
File modelFile = new File("en-pos-maxent.bin");
if (!modelFile.exists()) {
System.out.println("Downloading model.");
download("http://opennlp.sourceforge.net/models-1.5/en-pos-maxent.bin", modelFile);
}
POSModel model = new POSModel(modelFile);
PerformanceMonitor perfMon = new PerformanceMonitor(System.err, "sent");
POSTaggerME tagger = new POSTaggerME(model);
perfMon.start();
Stream.of(input).map(line -> {
String whitespaceTokenizerLine[] = WhitespaceTokenizer.INSTANCE.tokenize(line.toString());
String[] tags = tagger.tag(whitespaceTokenizerLine);
POSSample sample = new POSSample(whitespaceTokenizerLine, tags);
perfMon.incrementCounter();
return sample.toString();
}).forEach(System.out::println);
perfMon.stopAndPrintFinalResult();
}
}
This code doesn't actually test anything - it's a smoke test, if anything - but it should serve as a starting point. Another (potentially) nice thing is that it downloads a model for you if you don't have it downloaded already.
The URL http://bulba.sdsu.edu/jeanette/thesis/PennTags.html does not work anymore. I found the below on the 14th slide at http://www.slideshare.net/gagan1667/opennlp-demo
The above answer does provide a way to use the existing models from OpenNLP but if you need to train your own model, maybe the below can help:
Here is a detailed tutorial with full code:
https://dataturks.com/blog/opennlp-pos-tagger-training-java-example.php
Depending upon your domain, you can build a dataset either automatically or manually. Building such a dataset manually can be really painful, tools like POS tagger can help make the process much easier.
Training data format
Training data is passed as a text file where each line is one data item. Each word in the line should be labeled in a format like "word_LABEL", the word and the label name is separated by an underscore '_'.
anki_Brand overdrive_Brand
just_ModelName dance_ModelName 2018_ModelName
aoc_Brand 27"_ScreenSize monitor_Category
horizon_ModelName zero_ModelName dawn_ModelName
cm_Unknown 700_Unknown modem_Category
computer_Category
Train model
The important class here is POSModel, which holds the actual model. We use class POSTaggerME to do the model building. Below is the code to build a model from training data file
public POSModel train(String filepath) {
POSModel model = null;
TrainingParameters parameters = TrainingParameters.defaultParams();
parameters.put(TrainingParameters.ITERATIONS_PARAM, "100");
try {
try (InputStream dataIn = new FileInputStream(filepath)) {
ObjectStream<String> lineStream = new PlainTextByLineStream(new InputStreamFactory() {
#Override
public InputStream createInputStream() throws IOException {
return dataIn;
}
}, StandardCharsets.UTF_8);
ObjectStream<POSSample> sampleStream = new WordTagSampleStream(lineStream);
model = POSTaggerME.train("en", sampleStream, parameters, new POSTaggerFactory());
return model;
}
}
catch (Exception e) {
e.printStackTrace();
}
return null;
}
Use model to do tagging.
Finally, we can see how the model can be used to tag unseen queries:
public void doTagging(POSModel model, String input) {
input = input.trim();
POSTaggerME tagger = new POSTaggerME(model);
Sequence[] sequences = tagger.topKSequences(input.split(" "));
for (Sequence s : sequences) {
List<String> tags = s.getOutcomes();
System.out.println(Arrays.asList(input.split(" ")) +" =>" + tags);
}
}