We Keep Coding

Jaxb2Marshaller problem with String that contains '&'

I'm using the Jaxb2Marshaller from org.springframework.oxm.jaxb.Jaxb2Marshaller in my Spring Batch application to marshall XML with annotated classes. The implementation of the Marshaller is:
#Bean
public Jaxb2Marshaller productMarshaller() {
Map<String, Object> props = new HashMap<String, Object>();
props.put("com.sun.xml.bind.marshaller.CharacterEscapeHandler", new XmlCharacterEscapeHandler());
Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
marshaller.setClassesToBeBound(new Class[] {Product.class, TechSpecs.class});
marshaller.setMarshallerProperties(props);
return marshaller;
}
The Marshaller is used inside a StaxEventItemWriter that is implemented as following:
#Bean(name = "writer")
#StepScope
public StaxEventItemWriter<Product> writer (
#Value("#{jobParameters['path']}") String path,
#Value("#{stepExecutionContext['currentFile']}") String fileName
) {
Map<String, String> rootElementAttributes = new HashMap<String, String>();
rootElementAttributes.put("xmlns:xsi", "http://www.w3.org/2001/XMLSchema-instance");
FileSystemResource file = new FileSystemResource(path + fileName);
return new StaxEventItemWriterBuilder<Product>()
.name("writer")
.version("1.0")
.encoding("UTF-8")
.standalone(false)
.rootTagName("Products")
.rootElementAttributes(rootElementAttributes)
.headerCallback(headerCallback(null, null))
.footerCallback(footerCallback())
.marshaller(productMarshaller())
.resource(file)
.build();
}
Now the problem is that when I run the code, I get an IndexOutOfBoundsException. I found out that the exception is thrown because my Product object has a String attribute that may contain a &. The & is not allowed in XML and has to be escaped.
Why is the Jaxb2Marshaller not auto escaping the & character? As far as I understand the Marshaller should take care of escaping characters.
I tried to escape the character my self in the item processor with the StringEscapeUtils, e.g. product.setFullName(StringEscapeUtils.escapeXml10(dbExport.getFullName()));, but this didn't help. Also the String will be changed from & to &, which also contains a &.
I also tried to use my own implementation of a CharacterEscapeHandler, but the marshaller.setMarshallerProperties() does not have any visible effect on the Marshaller. Do I have to set the properties for the Marshaller differently?
public class XmlCharacterEscapeHandler implements CharacterEscapeHandler {
#Override
public void escape(char[] ch, int start, int length, boolean isAttVal, Writer out) throws IOException {
StringWriter buffer = new StringWriter();
for(int i = start; i < start + length; i++) {
buffer.write(ch[i]);
}
String escapedString = StringEscapeUtils.escapeXml10(buffer.toString());
out.write(escapedString);
}
}
EDIT
Unfortunately I could not resolve my issue until now. Therefore, I switched from Jaxb2Marshaller to XStreamMarshaller. Here I get a similar issue. As far as I can tell the underlying XStream should use a PrettyPrintWriter that will auto convert & to & as described here: https://stackoverflow.com/a/48141964/4191735 This is not happening. For me there is always an problem with &. Why does the escaping not work? Also escaping the String itself and force converting it to UTF-8 does not help.
Minimal Complete Example
Main:
package com.mwe;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.ComponentScan;
#SpringBootApplication
#ComponentScan("com.mwe")
public class Main {
public static void main(String [] args) {
System.exit(SpringApplication.exit(SpringApplication.run(Main.class, args)));
}
}
BatchConfig:
package com.mwe;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepScope;
import org.springframework.batch.item.ItemProcessor;
import org.springframework.batch.item.file.FlatFileItemReader;
import org.springframework.batch.item.file.mapping.DefaultLineMapper;
import org.springframework.batch.item.file.transform.DelimitedLineTokenizer;
import org.springframework.batch.item.xml.StaxEventItemWriter;
import org.springframework.batch.item.xml.builder.StaxEventItemWriterBuilder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.FileSystemResource;
import org.springframework.oxm.xstream.XStreamMarshaller;
#Configuration
#EnableBatchProcessing
public class BatchConfig {
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Bean
#StepScope
public FlatFileItemReader<Product> reader() {
FlatFileItemReader<Product> reader = new FlatFileItemReader<Product>();
reader.setResource(new FileSystemResource("test.csv"));
DefaultLineMapper<Product> lineMapper = new DefaultLineMapper<>();
lineMapper.setFieldSetMapper(new CustomFieldMapper());
DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();
tokenizer.setDelimiter("|");
tokenizer.setNames(new String[] {"ID", "NAME"});
lineMapper.setLineTokenizer(tokenizer);
reader.setLineMapper(lineMapper);
reader.setLinesToSkip(1);
return reader;
}
#Bean
public ItemProcessor<Product, Xml> processor() {
return new Processor();
}
#Bean
#StepScope
public StaxEventItemWriter<Xml> writer () {
return new StaxEventItemWriterBuilder<Xml>()
.name("writer")
.version("1.0")
.encoding("UTF-8")
.standalone(false)
.rootTagName("products")
.marshaller(getMarshaller())
.resource(new FileSystemResource("test.xml"))
.build();
}
#Bean
public Job job() {
return this.jobBuilderFactory.get("job")
.start(step1())
.build();
}
#Bean
public Step step1() {
return (stepBuilderFactory.get("step1")
.<Product, Xml>chunk(2)
.reader(reader())
.processor(processor())
.writer(writer())
.build());
}
#Bean
public XStreamMarshaller getMarshaller() {
XStreamMarshaller marshaller = new XStreamMarshaller();
marshaller.setEncoding("UTF-8");
return marshaller;
}
}
CustomFieldMapper
package com.mwe;
import org.springframework.batch.item.file.mapping.FieldSetMapper;
import org.springframework.batch.item.file.transform.FieldSet;
public class CustomFieldMapper implements FieldSetMapper<Product> {
public Product mapFieldSet(FieldSet fs) {
Product product = new Product();
product.setId(fs.readString("ID"));
product.setName(fs.readString("NAME"));
return product;
}
}
ItemProcessor:
package com.mwe;
import org.springframework.batch.item.ItemProcessor;
public class Processor implements ItemProcessor<Product, Xml> {
#Override
public Xml process(final Product product) {
Xml xml = new Xml();
xml.setId(Integer.parseInt(product.getId()));
xml.setName(product.getName());
return xml;
}
}
Product:
package com.mwe;
public class Product {
private String id;
private String name;
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
}
Xml:
package com.mwe;
public class Xml {
private int id;
private String name;
public int getId() {
return id;
}
public void setId(int id) {
this.id = id;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
}
Application Properties:
# Spring config
spring.main.allow-bean-definition-overriding=true
spring.main.banner-mode=off
spring.batch.initialize-schema=never
# Logging data source
spring.datasource.logging.driver-class-name=org.mariadb.jdbc.Driver
spring.datasource.logging.maximum-pool-size=10
spring.datasource.logging.hikar.minimum-idle=1
spring.datasource.logging.hikari.data-source-properties.useUnicode=true
spring.datasource.logging.hikari.data-source-properties.characterEncoding=UTF-8
spring.datasource.logging.hibernate.dialect=org.hibernate.dialect.MariaDBDialect
spring.datasource.logging.hibernate.ddl-auto=none
spring.datasource.url=jdbc:mariadb://localhost:3306/logging?UseUnicode=true&characterEncoding=utf8
spring.datasource.username=root
spring.datasource.password=root
Pom:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.4.2</version>
<relativePath /> <!-- lookup parent from repository -->
</parent>
<groupId>com.mwe</groupId>
<artifactId>mwe</artifactId>
<version>1</version>
<name>mwe</name>
<description>Minimal working example</description>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-batch</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.mariadb.jdbc</groupId>
<artifactId>mariadb-java-client</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-oxm</artifactId>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-text</artifactId>
<version>1.9</version>
</dependency>
<dependency>
<groupId>javax.activation</groupId>
<artifactId>activation</artifactId>
<version>1.1.1</version>
</dependency>
<dependency>
<groupId>com.thoughtworks.xstream</groupId>
<artifactId>xstream</artifactId>
<version>1.4.15</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
test.csv:
"ID"|"NAME"
1|"Product 1"
2|"Product 1 & Addition"

You are hitting this issue: https://github.com/spring-projects/spring-batch/issues/3745.
This issue will be fixed in v4.3.2/v4.2.6 which are planned to be released on March 18, 2021. Please check the milestone page on GitHub in case the release date changes.

picocli does not work with jline3 in cmd.exe

I want to use picocli with jline3. So I create a project with the following pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.flaxel</groupId>
<artifactId>picocli_test</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>picocli</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
<picocli.version>3.9.3</picocli.version>
<jline.version>3.9.0</jline.version>
</properties>
<dependencies>
<dependency>
<groupId>info.picocli</groupId>
<artifactId>picocli</artifactId>
<version>${picocli.version}</version>
</dependency>
<dependency>
<groupId>info.picocli</groupId>
<artifactId>picocli-shell-jline3</artifactId>
<version>${picocli.version}</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>3.0.2</version>
<configuration>
<archive>
<manifest>
<mainClass>com.flaxel.picocli.App</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compile-plugin</artifactId>
<version>3.7.0</version>
</plugin>
</plugins>
</build>
Now I have copied a class from the picocli github page:
package com.flaxel.picocli;
import java.io.IOException;
import java.io.PrintWriter;
import java.nio.charset.Charset;
import java.util.concurrent.Callable;
import java.util.concurrent.TimeUnit;
import org.jline.reader.EndOfFileException;
import org.jline.reader.LineReader;
import org.jline.reader.LineReaderBuilder;
import org.jline.reader.MaskingCallback;
import org.jline.reader.ParsedLine;
import org.jline.reader.UserInterruptException;
import org.jline.reader.impl.DefaultParser;
import org.jline.reader.impl.LineReaderImpl;
import org.jline.terminal.Terminal;
import org.jline.terminal.TerminalBuilder;
import picocli.CommandLine;
import picocli.CommandLine.Command;
import picocli.CommandLine.Option;
import picocli.CommandLine.ParentCommand;
import picocli.shell.jline3.PicocliJLineCompleter;
/**
* Example that demonstrates how to build an interactive shell with JLine3 and
* picocli.
*
* #since 3.9
*/
public class App {
/**
* Top-level command that just prints help.
*/
#Command(name = "", description = "Example interactive shell with completion", footer = { "",
"Press Ctl-D to exit." }, subcommands = { MyCommand.class, ClearScreen.class })
static class CliCommands implements Runnable {
LineReaderImpl reader;
PrintWriter out;
CliCommands() {
}
public void setReader(LineReader reader) {
this.reader = (LineReaderImpl) reader;
out = reader.getTerminal().writer();
}
#Override
public void run() {
out.println(new CommandLine(this).getUsageMessage());
}
}
/**
* A command with some options to demonstrate completion.
*/
#Command(name = "cmd", mixinStandardHelpOptions = true, version = "1.0", description = "Command with some options to demonstrate TAB-completion"
+ " (note that enum values also get completed)")
static class MyCommand implements Runnable {
#Option(names = { "-v", "--verbose" })
private boolean[] verbosity = {};
#Option(names = { "-d", "--duration" })
private int amount;
#Option(names = { "-u", "--timeUnit" })
private TimeUnit unit;
#ParentCommand
CliCommands parent;
#Override
public void run() {
if (verbosity.length > 0) {
parent.out.printf("Hi there. You asked for %d %s.%n", amount, unit);
} else {
parent.out.println("hi!");
}
}
}
/**
* Command that clears the screen.
*/
#Command(name = "cls", aliases = "clear", mixinStandardHelpOptions = true, description = "Clears the screen", version = "1.0")
static class ClearScreen implements Callable<Void> {
#ParentCommand
CliCommands parent;
#Override
public Void call() throws IOException {
parent.reader.clearScreen();
return null;
}
}
public static void main(String[] args) {
try {
// set up the completion
CliCommands commands = new CliCommands();
CommandLine cmd = new CommandLine(commands);
Terminal terminal = TerminalBuilder.builder().build();
LineReader reader = LineReaderBuilder.builder()
.terminal(terminal)
.completer(new PicocliJLineCompleter(cmd.getCommandSpec()))
.parser(new DefaultParser())
.build();
commands.setReader(reader);
String prompt = "prompt> ";
String rightPrompt = null;
// start the shell and process input until the user quits with Ctrl-D
String line;
while (true) {
try {
line = reader.readLine(prompt, rightPrompt, (MaskingCallback) null, null);
ParsedLine pl = reader.getParser().parse(line, 0);
String[] arguments = pl.words().toArray(new String[0]);
CommandLine.run(commands, arguments);
} catch (UserInterruptException e) {
// Ignore
} catch (EndOfFileException e) {
return;
}
}
} catch (Throwable t) {
t.printStackTrace();
}
}
}
If I run the code in the Eclipse IDE, I can write a command in the console and I get an answer. But if I create a jar file with maven, it does not work. I get the following error:
Error: Unable to initialize main class com.flaxel.picocli.App
Caused by: java.lang.NoClassDefFoundError: org/jline/reader/UserInterruptException
I work with Eclipse 2018-09 and Java 11.

The NoClassDefFoundError occurs because you are missing the JLine classes when running your jar. Try using the shade Maven plugin to create an Uber-jar that contains all the dependencies.
With JLine 3 on Windows, you probably want to add the org.jline:jline-terminal-jansi:3.9.0 dependency in addition to info.picoli:picocli-shell-jline3:3.9.3. Without this (or the jline-terminal-jna dependency) you will get a “dumb” terminal without ANSI colors or autocompletion.
(info.picoli:picocli-shell-jline3:3.9.3 will bring in info.picocli:picocli and org.jline:jline as transitive dependencies, so there is no need to include these explicitly.)

Perfect that works for me. I use the maven shade plugin and add some properties, so I get this pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.flaxel</groupId>
<artifactId>picocli_test</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>picocli</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
<picocli.version>3.9.3</picocli.version>
<jline.version>3.9.0</jline.version>
</properties>
<dependencies>
<dependency>
<groupId>info.picocli</groupId>
<artifactId>picocli</artifactId>
<version>${picocli.version}</version>
</dependency>
<dependency>
<groupId>info.picocli</groupId>
<artifactId>picocli-shell-jline3</artifactId>
<version>${picocli.version}</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>com.flaxel.picocli.App</mainClass>
<addDefaultImplementationEntries>true</addDefaultImplementationEntries>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compile-plugin</artifactId>
<version>3.7.0</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
<shadedArtifactAttached>true</shadedArtifactAttached>
<shadedClassifierName>jar-with-dependencies</shadedClassifierName>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
EDIT
I have at least one question. If I start the application and want to use the tab completion, it does not work. So I get no completion, only four spaces are set:
Feb. 07, 2019 6:50:29 NACHM. org.jline.utils.Log logr
WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)
prompt> cmd
Or did I something wrong?

AWS Polly Java example

I am trying to execute a simple Java example that uses the AWS Polly service. I am using the code provided by AWS on their documentation. I created a simple Maven Project using the following -**
1. group id - com.amazonaws.polly
2. artifact id - java-demo
3. version - 0.0.1-SNAPSHOT
Following is my project structure -
Following is my pom.xml -
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.amazonaws.polly</groupId>
<artifactId>java-demo</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependencies>
<!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-polly -->
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-polly</artifactId>
<version>1.11.77</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.googlecode.soundlibs/jlayer -->
<dependency>
<groupId>com.googlecode.soundlibs</groupId>
<artifactId>jlayer</artifactId>
<version>1.0.1-1</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.2.1</version>
<executions>
<execution>
<goals>
<goal>java</goal>
</goals>
</execution>
</executions>
<configuration>
<mainClass>com.amazonaws.demos.polly.PollyDemo</mainClass>
</configuration>
</plugin>
</plugins>
</build>
</project>
Following is my java class -
package com.amazonaws.demos.polly;
import java.io.IOException;
import java.io.InputStream;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
import com.amazonaws.regions.Region;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.polly.AmazonPollyClient;
import com.amazonaws.services.polly.model.DescribeVoicesRequest;
import com.amazonaws.services.polly.model.DescribeVoicesResult;
import com.amazonaws.services.polly.model.OutputFormat;
import com.amazonaws.services.polly.model.SynthesizeSpeechRequest;
import com.amazonaws.services.polly.model.SynthesizeSpeechResult;
import com.amazonaws.services.polly.model.Voice;
import javazoom.jl.player.advanced.AdvancedPlayer;
import javazoom.jl.player.advanced.PlaybackEvent;
import javazoom.jl.player.advanced.PlaybackListener;
public class PollyDemo {
private final AmazonPollyClient polly;
private final Voice voice;
private static final String SAMPLE = "Congratulations. You have successfully built this working demo "+
"of Amazon Polly in Java. Have fun building voice enabled apps with Amazon Polly (that's me!), and always"+
"look at the AWS website for tips and tricks on using Amazon Polly and other great services from AWS";
public PollyDemo(Region region) {
//Didn't work
//AWSCredentials credentials = new BasicAWSCredentials("someAccessKey","someSecretKey");
//polly = new AmazonPollyClient(credentials);
//Didn't work
// create an Amazon Polly client in a specific region
polly = new AmazonPollyClient(new DefaultAWSCredentialsProviderChain(),
new ClientConfiguration());
polly.setRegion(region);
// Create describe voices request.
DescribeVoicesRequest describeVoicesRequest = new DescribeVoicesRequest();
// Synchronously ask Amazon Polly to describe available TTS voices.
DescribeVoicesResult describeVoicesResult = polly.describeVoices(describeVoicesRequest);
voice = describeVoicesResult.getVoices().get(0);
}
public InputStream synthesize(String text, OutputFormat format) throws IOException {
SynthesizeSpeechRequest synthReq =
new SynthesizeSpeechRequest().withText(text).withVoiceId(voice.getId())
.withOutputFormat(format);
SynthesizeSpeechResult synthRes = polly.synthesizeSpeech(synthReq);
return synthRes.getAudioStream();
}
public static void main(String args[]) throws Exception {
//create the test class
PollyDemo helloWorld = new PollyDemo(Region.getRegion(Regions.US_EAST_1));
//get the audio stream
InputStream speechStream = helloWorld.synthesize(SAMPLE, OutputFormat.Mp3);
//create an MP3 player
AdvancedPlayer player = new AdvancedPlayer(speechStream,
javazoom.jl.player.FactoryRegistry.systemRegistry().createAudioDevice());
player.setPlayBackListener(new PlaybackListener() {
#Override
public void playbackStarted(PlaybackEvent evt) {
System.out.println("Playback started");
System.out.println(SAMPLE);
}
#Override
public void playbackFinished(PlaybackEvent evt) {
System.out.println("Playback finished");
}
});
// play it!
player.play();
}
}
I am running the code locally, therefore I have my AWS IAM credentials configured in my system,
My IAM user also has access to AWS Polly service.
I am getting the following error when I run the code -
Exception in thread "main" com.amazonaws.services.polly.model.AmazonPollyException: The security token included in the request is invalid. (Service: AmazonPolly; Status Code: 403; Error Code: UnrecognizedClientException; Request ID: 4d4b01fb-8015-11e8-8e18-4548f95fba92)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1586)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1254)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1035)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:747)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:721)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:704)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:672)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:654)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:518)
at com.amazonaws.services.polly.AmazonPollyClient.doInvoke(AmazonPollyClient.java:668)
at com.amazonaws.services.polly.AmazonPollyClient.invoke(AmazonPollyClient.java:644)
at com.amazonaws.services.polly.AmazonPollyClient.describeVoices(AmazonPollyClient.java:383)
at com.amazonaws.demos.polly.PollyDemo.<init>(PollyDemo.java:39)
at com.amazonaws.demos.polly.PollyDemo.main(PollyDemo.java:54)
I am referring the following AWS Doc for the Polly java example-
https://docs.aws.amazon.com/polly/latest/dg/examples-java.html
Can someone help fix my code? What do I change in my code?

It's a 403 error. Where are you passing the AWS access and secret key? You can try this
* Constructs a new client to invoke service methods on AmazonPolly. A
* credentials provider chain will be used that searches for credentials in
* this order:
* <ul>
* <li>Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY</li>
* <li>Java System Properties - aws.accessKeyId and aws.secretKey</li>
* <li>Instance profile credentials delivered through the Amazon EC2
* metadata service</li>
* </ul>
* <p>
* All service calls made using this new client object are blocking, and
* will not return until the service call completes.
*
* #see DefaultAWSCredentialsProviderChain
*/
public AmazonPollyClient() {
this(new DefaultAWSCredentialsProviderChain(), new ClientConfiguration());
}
https://github.com/aws/aws-sdk-android/blob/master/aws-android-sdk-polly/src/main/java/com/amazonaws/services/polly/AmazonPollyClient.java

Replace
new DefaultAWSCredentialsProviderChain()
with
AWSStaticCredentialsProvider(new BasicAWSCredentials("AccessKey", "Secret Key"))

I noticed there is no POM file in the Github repository for this service. We will fix that. Here is a POM file that works.
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>pollyJ1Project</groupId>
<artifactId>pollyJ1Project</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<java.version>1.8</java.version>
</properties>
<dependencies>
<!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-polly -->
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-polly</artifactId>
<version>1.11.774</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.awaitility/awaitility -->
<!-- https://mvnrepository.com/artifact/org.awaitility/awaitility -->
<dependency>
<groupId>org.awaitility</groupId>
<artifactId>awaitility</artifactId>
<version>4.0.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.awaitility</groupId>
<artifactId>awaitility</artifactId>
<version>4.0.2</version>
<scope>compile</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build>
Also - here is an example you can try:
package com.amazonaws.polly.samples;
import com.amazonaws.services.polly.AmazonPolly;
import com.amazonaws.services.polly.AmazonPollyClientBuilder;
import com.amazonaws.services.polly.model.DescribeVoicesRequest;
import com.amazonaws.services.polly.model.DescribeVoicesResult;
public class DescribeVoicesSample {
public static void main(String[] args) {
AmazonPolly client = AmazonPollyClientBuilder.defaultClient();
describeVoices(client);
}
public static void describeVoices(AmazonPolly client ) {
DescribeVoicesRequest allVoicesRequest = new DescribeVoicesRequest();
DescribeVoicesRequest enUsVoicesRequest = new DescribeVoicesRequest()
.withLanguageCode("en-US");
try {
String nextToken;
do {
DescribeVoicesResult allVoicesResult =
client.describeVoices(allVoicesRequest);
nextToken = allVoicesResult.getNextToken();
allVoicesRequest.setNextToken(nextToken);
System.out.println("All voices: " + allVoicesResult.getVoices());
} while (nextToken != null);
do {
DescribeVoicesResult enUsVoicesResult = client.describeVoices(enUsVoicesRequest);
nextToken = enUsVoicesResult.getNextToken();
enUsVoicesRequest.setNextToken(nextToken);
System.out.println("en-US voices: " + enUsVoicesResult.getVoices());
} while (nextToken != null);
} catch (Exception e) {
System.err.println("Exception caught: " + e);
}
}
}
The above code is V1. You can find V2 code examples here:
https://github.com/awsdocs/aws-doc-sdk-examples/tree/master/javav2/example_code/polly

I found a nice working demo here https://youtu.be/WMMSQAn_vHI. It is working java example for AWS Polly service using new DefaultAWSCredentialsProviderChain() only.

Could not find an appropriate constructor for com.google.cloud.bigtable.hbase1_x.BigtableConnection

I tried to create a simple class to create a table and add some columns with HBase and Google app engine.
I have already created a project and an instance in Google Cloud platform.
I cloned this repository : https://github.com/GoogleCloudPlatform/cloud-bigtable-examples/blob/master/java/hello-world/src/main/java/com/example/cloud/bigtable/helloworld/HelloWorld.java
It works like a charm to create a table in my instance.
But when I'm trying to create a new maven project with the same config, it doesn't work, I can't create anything.
I got this issue :
InvocationTargetException: Could not find an appropriate constructor for com.google.cloud.bigtable.hbase1_x.BigtableConnection: com.google.common.util.concurrent.MoreExecutors.platformThreadFactory()Ljava/util/concurrent/ThreadFactory;
here is my AudioBridgeData.java file :
package com.xxx.xxx;
import com.google.cloud.bigtable.hbase.BigtableConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.util.Bytes;
import java.io.IOException;
/**
* A minimal application that connects to Cloud Bigtable using the native HBase API
* and performs some basic operations.
*/
public class AudioBridgeData {
private static final byte[] TABLE_NAME = Bytes.toBytes("audio-bridge");
private static final byte[] COLUMN_FAMILY_NAME = Bytes.toBytes("phone-number");
private static final byte[] COLUMN_NAME = Bytes.toBytes("number");
private static final String[] NUMBERS =
{ "+33697584976", "+19553560976", "+4879665676" };
/**
* Connects to Cloud Bigtable, runs some basic operations and prints the results.
*/
private static void doAudioBridge(String projectId, String instanceId) {
try (Connection connection = BigtableConfiguration.connect(projectId, instanceId)) {
Admin admin = connection.getAdmin();
HTableDescriptor descriptor = new HTableDescriptor(TableName.valueOf(TABLE_NAME));
descriptor.addFamily(new HColumnDescriptor(COLUMN_FAMILY_NAME));
print("Create table " + descriptor.getNameAsString());
admin.createTable(descriptor);
Table table = connection.getTable(TableName.valueOf(TABLE_NAME));
print("Write some numbers to the table");
for (int i = 0; i < NUMBERS.length; i++) {
Put put = new Put(Bytes.toBytes(i));
put.addColumn(COLUMN_FAMILY_NAME, COLUMN_NAME, Bytes.toBytes(NUMBERS[i]));
table.put(put);
}
int rowKey = 0;
Result getResult = table.get(new Get(Bytes.toBytes(rowKey)));
String number = Bytes.toString(getResult.getValue(COLUMN_FAMILY_NAME, COLUMN_NAME));
System.out.println("Get a single number by row key");
System.out.printf("\t%s = %s\n", rowKey, number);
Scan scan = new Scan();
print("Scan for all numbers:");
ResultScanner scanner = table.getScanner(scan);
for (Result row : scanner) {
byte[] valueBytes = row.getValue(COLUMN_FAMILY_NAME, COLUMN_NAME);
System.out.println('\t' + Bytes.toString(valueBytes));
}
} catch (IOException e) {
System.err.println("Exception while running HelloWorld: " + e.getMessage());
e.printStackTrace();
System.exit(1);
}
System.exit(0);
}
private static void print(String msg) {
System.out.println("Number: " + msg);
}
public static void main(String[] args) {
String projectId = requiredProperty("bigtable.projectID");
String instanceId = requiredProperty("bigtable.instanceID");
doAudioBridge(projectId, instanceId);
}
private static String requiredProperty(String prop) {
String value = System.getProperty(prop);
if (value == null) {
throw new IllegalArgumentException("Missing required system property: " + prop);
}
return value;
}
}
Here is my pom.xml file :
<?xml version="1.0" encoding="UTF-8"?>
<project>
<modelVersion>4.0.0</modelVersion>
<groupId>com.xxx</groupId>
<artifactId>xxx</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<url>http://maven.apache.org</url>
<name>xxx</name>
<properties>
<bigtable.version>1.0.0-pre1</bigtable.version>
<hbase.version>1.1.5</hbase.version>
<maven.compiler.target>1.8</maven.compiler.target>
<maven.compiler.source>1.8</maven.compiler.source>
</properties>
<repositories>
<repository>
<id>snapshots-repo</id>
<url>https://oss.sonatype.org/content/repositories/snapshots</url>
<releases><enabled>false</enabled></releases>
<snapshots><enabled>true</enabled></snapshots>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>com.google.cloud.bigtable</groupId>
<artifactId>bigtable-hbase-1.x</artifactId>
<version>${bigtable.version}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<version>1.1.33.Fork26</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>0.98.11-hadoop2</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.5.0</version>
<configuration>
<mainClass>com.xxx.xxx.AudioBridgeData</mainClass>
</configuration>
</plugin>
</plugins>
</build>
</project>
And I tried to run this command :
mvn exec:java -Dbigtable.projectID=xxx -Dbigtable.instanceID=quickstart-instance
Thanks a lot for your help ! :)

I had the same error message, weirdly it was caused by missing application credentials. If its the same issue you should set this environment variable:
GOOGLE_APPLICATION_CREDENTIALS
It should be set to the location of the client credentials file you (might have) downloaded after creating a service account key. This is a good page:
https://developers.google.com/identity/protocols/application-default-credentials

Kafka: Cant Create Multiple Stream Consumers

I just got up and running with Kafka 0.8 beta 1. I have a really simple example up and running, the problem is, I can only get one message consumer to work, not several. That is, the runSingleWorker() method WORKS. The run() method DOES NOT WORK:
import kafka.consumer.ConsumerIterator;
import kafka.consumer.KafkaStream;
import kafka.consumer.ConsumerConfig;
import kafka.javaapi.consumer.ConsumerConnector;
import java.util.Map;
import java.util.List;
import java.util.HashMap;
import java.util.concurrent.Executors;
import java.util.concurrent.ExecutorService;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import com.truecar.inventory.worker.core.application.config.AppConfig;
public class ConsumerThreadPool {
private final ConsumerConnector consumer;
private final String topic;
private ExecutorService executor;
private static ApplicationContext context = new AnnotationConfigApplicationContext(AppConfig.class);
public ConsumerThreadPool(String topic) {
consumer = kafka.consumer.Consumer.createJavaConsumerConnector((ConsumerConfig)context.getBean("consumerConfig"));
this.topic = topic;
}
public void shutdown() {
if (consumer != null) consumer.shutdown();
if (executor != null) executor.shutdown();
}
public void run(Integer numThreads) {
Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
topicCountMap.put(topic, numThreads);
Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.createMessageStreams(topicCountMap);
List<KafkaStream<byte[], byte[]>> topicListeners = consumerMap.get(topic);
executor = Executors.newFixedThreadPool(numThreads);
for(Integer i = 0; i < numThreads; i++ ){
KafkaStream<byte[], byte[]> stream = topicListeners.get(i);
executor.submit(new Consumer(stream, i));
}
}
public void runSingleWorker(Integer numThreads) {
Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
topicCountMap.put(topic, new Integer(1));
Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.createMessageStreams(topicCountMap);
KafkaStream<byte[], byte[]> stream = consumerMap.get(topic).get(0);
ConsumerIterator<byte[], byte[]> it = stream.iterator();
while(true) {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
while(it.hasNext()){
System.out.println(new String(it.next().message()));
}
}
}
}
And inside my toy consumer:
import kafka.consumer.KafkaStream;
import kafka.consumer.ConsumerIterator;
public class Consumer implements Runnable {
private KafkaStream kafkaStream;
private Integer threadNumber;
public Consumer(KafkaStream kafkaStream, Integer threadNumber) {
this.threadNumber = threadNumber;
this.kafkaStream = kafkaStream;
}
public void run() {
ConsumerIterator<byte[], byte[]> it = kafkaStream.iterator();
System.out.println("Created iterator " + it.toString() + " thread number " + threadNumber);
while(true) {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
break;
}
while(it.hasNext()) {
System.out.println("Thread " + threadNumber + ": " + new String(it.next().message()));
}
}
System.out.println("Shutting down Thread: " + threadNumber);
}
}
The problem is, the pool of workers does not pick up messages:
Created iterator empty iterator thread number 3
Created iterator empty iterator thread number 6
Created iterator empty iterator thread number 9
Created iterator empty iterator thread number 7
Created iterator empty iterator thread number 0
Created iterator empty iterator thread number 0
Created iterator empty iterator thread number 8
Created iterator empty iterator thread number 3
etc...
When I add messages via the produce command line, the messages are printed under the single threaded worker version, but messages are not printed under the multi-stream situation. Whats going on here? How can I fix this?
Btw, the pom.xml for kafka 0.8 is not a valid pom and will not acquire dependencies, so here is a pom with complete dependencies.
<?xml version="1.0" encoding="UTF-8"?>
<project
xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>group1</groupId>
<artifactId>artifact1</artifactId>
<version>0.1.0</version>
<packaging>jar</packaging>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<org.springframework.version>3.2.4.RELEASE</org.springframework.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
<version>3.2.4.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
<version>3.2.4.RELEASE</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.9.2</artifactId>
<version>0.8.0-beta1</version>
</dependency>
<dependency>
<groupId>javax.inject</groupId>
<artifactId>javax.inject</artifactId>
<version>1</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.9.2</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
<dependency>
<groupId>com.101tec</groupId>
<artifactId>zkclient</artifactId>
<version>0.3</version>
</dependency>
<dependency>
<groupId>com.yammer.metrics</groupId>
<artifactId>metrics-core</artifactId>
<version>2.2.0</version>
</dependency>
</dependencies>
<build>
<finalName>inventory-core</finalName>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.0</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>com.truecar.inventory.worker.core.application.Starter</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.dstovall</groupId>
<artifactId>onejar-maven-plugin</artifactId>
<version>1.4.4</version>
<executions>
<execution>
<configuration>
<onejarVersion>0.97</onejarVersion>
<classifier>onejar</classifier>
</configuration>
<goals>
<goal>one-jar</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<pluginRepositories>
<pluginRepository>
<id>onejar-maven-plugin.googlecode.com</id>
<url>http://onejar-maven-plugin.googlecode.com/svn/mavenrepo</url>
</pluginRepository>
</pluginRepositories>
</project>

Maybe too late for questioner but could be useful to other developers.
Seems you have used only one Partition for couple of Consumers – that's wrong.
Quote from Documentation:
Since there are many partitions this still balances the load over many consumer instances. Note however that there cannot be more consumer instances than partitions.
So when you think about Consumers you should think how to divide messages by partitions.
In most cases you should use some high-level grouping to it or even let it be random by default.