When I'm trying to run my code it throws this Exception:
Exception in thread "main" org.apache.spark.SparkException: Could not parse Master URL:spark:http://localhost:18080
This is my code:
SparkConf conf = new SparkConf().setAppName("App_Name").setMaster("spark:http://localhost:18080").set("spark.ui.port","18080");
JavaStreamingContext ssc = new JavaStreamingContext(sc, new Duration(1000));
String[] filet=new String[]{"Obama","ISI"};
JavaReceiverInputDStream<Status> reciverStream=TwitterUtils.createStream(ssc,filet);
JavaDStream<String> statuses = reciverStream.map(new Function<Status, String>() {
public String call(Status status) { return status.getText(); }
}
);
ssc.start();
ssc.awaitTermination();}}
Any idea how can I fix this problem?
The problem is that you specify 2 schemas in the URL you pass to SparkConf.setMaster().
The spark is the schema, so you don't need to add http after spark. See the javadoc of SparkConf.setMaster() for more examples.
So the master URL you should be using is "spark://localhost:18080". Change this line:
SparkConf conf = new SparkConf().setAppName("App_Name")
.setMaster("spark://localhost:18080").set("spark.ui.port","18080");
The standard port for master is 7077 not 18080. Maybe you can try 7077.
Related
//I want to add configuration to input topic
public void createTopics(Properties envProps) {
Map<String, Object> config = new HashMap<>();
config.put("bootstrap.servers", envProps.getProperty("bootstrap.servers"));
AdminClient client = AdminClient.create(config);
List<NewTopic> topics = new ArrayList<>();
topics.add(new NewTopic(
envProps.getProperty("input.topic.name") ,
Integer.parseInt(envProps.getProperty("input.topic.partitions")),
Short.parseShort(envProps.getProperty("input.topic.replication.factor"))));
topics.add(new NewTopic(
envProps.getProperty("output.topic.name"),
Integer.parseInt(envProps.getProperty("output.topic.partitions")),
Short.parseShort(envProps.getProperty("output.topic.replication.factor"))));
client.createTopics(topics);
client.close();
}
public Properties loadEnvProperties(String fileName) throws IOException {
Properties envProps = new Properties();
FileInputStream input = new FileInputStream(fileName);
envProps.load(input);
input.close();
return envProps;
}
I am able to create the topics. But, I want to configure topic in this code ie., cleanup.policy. How can I incorporate this here?
You can find the source code for kafka-topics --create here
As you can see, it calls NewTopic#configs with a Map
In that map, you would have (cleanup.policy, compact)
You could also call TopicCommand.main on your own, rather than re-write most of the existing code.
I have already uploaded a fat jar containing my application code to the /lib folder of all nodes in my Flink cluster. I am trying to start the Flink job from a separate java application, but can't find a good way to do so.
The closest thing to a solution that I have currently found is the Monitoring Rest API which has a run job API. However, this only allows you to run jobs submitted via the job upload function.
I have seen the ClusterClient.java in the flink-client module, but could not see any examples of how I might use this.
Any examples of how someone has submitted jobs successfully through java code would be greatly appreciated!
You can use RestClusterClient to run a PackagedProgram which points to your Flink job. If your job accepts some arguments, you can pass them.
Here is an example for a standalone cluster running on localhost:8081 :
// import org.apache.flink.api.common.JobSubmissionResult;
// import org.apache.flink.client.deployment.StandaloneClusterId;
// import org.apache.flink.client.program.PackagedProgram;
// import org.apache.flink.client.program.rest.RestClusterClient;
// import org.apache.flink.configuration.Configuration;
// import org.apache.flink.configuration.JobManagerOptions;
// import org.apache.flink.configuration.RestOptions;
String clusterHost = "localhost";
int clusterPort = 8081;
Configuration config = new Configuration();
config.setString(JobManagerOptions.ADDRESS, clusterHost);
config.setInteger(RestOptions.PORT, clusterPort);
String jarFilePath = "/opt/flink/examples/streaming/SocketWindowWordCount.jar";
String[] args = new String[]{ "--port", "9000" };
PackagedProgram packagedProgram = new PackagedProgram(new File(jarFilePath), args);
RestClusterClient<StandaloneClusterId> client =
new RestClusterClient<StandaloneClusterId>(config, StandaloneClusterId.getInstance());
int parallelism = 1;
JobSubmissionResult result = client.run(packagedProgram, parallelism);
This seems to work for version 1.10
private static final int PARALLELISM = 8;
private static final Configuration FLINK_CONFIG = new Configuration();
void foo() throws Exception {
FLINK_CONFIG.setString(JobManagerOptions.ADDRESS, "localhost");
FLINK_CONFIG.setInteger(RestOptions.PORT, 8081);
FLINK_CONFIG.setInteger(RestOptions.RETRY_MAX_ATTEMPTS, 3);
RestClusterClient<StandaloneClusterId> flinkClient = new RestClusterClient<>(FLINK_CONFIG, StandaloneClusterId.getInstance());
String jar = "/path/to/jar";
String[] args = new String[]{"..."};
PackagedProgram program = PackagedProgram.newBuilder()
.setJarFile(new File(jar))
.setArguments(args)
.build();
JobGraph jobGraph = PackagedProgramUtils.createJobGraph(program, FLINK_CONFIG, PARALLELISM, false);
JobID jobId = flinkClient.submitJob(jobGraph).get();
...
}
I am running a storm topology .This is the basic wordcount topology.I am using text file as the source and storm for processing the data.While submitting the i am facing these issues.I am very new to storm.Please suggest me the changes i need to do in the following code.Thanks in Advance !!
My topology
public class TopologyMain {
public static void main(String[] args) throws InterruptedException, AlreadyAliveException, InvalidTopologyException {
//Topology definition
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("word-reader",new WordReader());
builder.setBolt("word-normalizer", new WordNormalizer())
.shuffleGrouping("word-reader");
builder.setBolt("word-counter", new WordCounter(),1)
.fieldsGrouping("word-normalizer", new Fields("word"));
//Configuration
Config conf = new Config();
conf.put("wordsFile", args[0]);
conf.setDebug(false);
//Topology run
conf.put(Config.TOPOLOGY_MAX_SPOUT_PENDING, 1);
conf.put(Config.NIMBUS_HOST, "192.168.1.229");
//LocalCluster cluster = new LocalCluster();
//cluster.submitTopology("Getting-Started-Toplogie", conf, builder.createTopology());
//Thread.sleep(1000);
System.setProperty("storm.jar", "/home/raremile/st/examples-ch02-getting_started/target/Getting-Started-0.0.1-SNAPSHOT.jar");
StormSubmitter.submitTopology("Count-Word-Topology-With-Refresh-Cache", conf,
builder.createTopology());
//cluster.shutdown();
}
}
ERROR
Exception in thread "main" java.lang.NoSuchMethodError: backtype.storm.topology.TopologyBuilder.setBolt(Ljava/lang/String;Lbacktype/storm/topology/IBasicBolt;Ljava/lang/Integer;)Lbacktype/storm/topology/BoltDeclarer;
at TopologyMain.main(TopologyMain.java:21)
I am able to run this code in local mode without any error.
changed the version to the 0.9.0.1 and i am able to run it
<dependency>
<groupId>storm</groupId>
<artifactId>storm</artifactId>
<version>0.9.0.1</version>
</dependency>
I am new to opennlp, I am getting Missing the manifest.properties! exception when i excute the following code,please tell me suggestion to avoid this.
public class PrePostProcessing_Peregrine {
public Map<String,Set<String>> btntMap;
public Map<String, String> fishMap;
public SentenceModel sModel;
public SentenceDetectorME sentDet;
public Map<String,Set<String>> topBottomTermSet;
public PrePostProcessing_Peregrine() throws IOException {
FileInputStream str=new FileInputStream("/home/rajendraprasad.yk/Desktop/data/en-sent.bin");
System.out.println(str+"===================>");
SentenceModel sModel = new SentenceModel(str);
System.out.println("===================model =================>"+sModel);
sentDet = new SentenceDetectorME(sModel);
System.out.println("===================>sentDet "+sentDet);
System.err.println("Sentence Detector Initialized");
Exception is:
opennlp.tools.util.InvalidFormatException: Missing the manifest.properties!
at opennlp.tools.util.model.BaseModel.validateArtifactMap(BaseModel.java:217)
at opennlp.tools.sentdetect.SentenceModel.validateArtifactMap(SentenceModel.java:78)
at opennlp.tools.util.model.BaseModel.<init>(BaseModel.java:142)
at opennlp.tools.sentdetect.SentenceModel.<init>(SentenceModel.java:73)
at com.molcon.Text_Mining.PrePostProcessing_Peregrine.<init>(PrePostProcessing_Peregrine.java:66)
at com.molcon.Text_Mining.TextMining.peregrineRun(TextMining.java:207)
at com.molcon.Text_Mining.TextMining.process_journals_for_Mining(TextMining.java:108)
I made mistake at FileInputStream modelIn = new FileInputStream("/home/rajendraprasad.yk/Desktop/data/en-sent.bin"); now I changed to InputStream modelIn = new FileInputStream("/home/rajendraprasad.yk/Desktop/data/en-sent.bin"); from this changes am not getting any exception but not able to load file from SentenceModel.
when i execute this line SentenceModel sModel = new SentenceModel(modelIn); am not getting any response,please help
For what I know there are two possible causes:
the model you are using is corrupted, try to download it again
the version of the model and of the OpenNLP library that you are using are not matching. As I read from the official website models are version specific, so you should try to understand if this is you case, and act accordingly.
If you check this constructor you will see that manifest.properties is not a file, it is a set of hard-coded properties:
Properties manifest = new Properties();
manifest.setProperty(MANIFEST_VERSION_PROPERTY, "1.0");
...
artifactMap.put("manifest.properties", manifest);
When you compare it to the InputStream constructor you can see that manifest.properties is nowhere to be found, only loadModel(in) which leads to conclusion that manifest.properties should be in the model file.
Why this works in a standalone application and not inside Tomcat requires some debugging.
As #5agado suggested, your model's version might be different than the library's.
Good afternoon all!
I use PayPal REST API java sdk and I want to have different configurations for different environments of my application. Here is how I'm trying to do so:
private static boolean IS_PRODUCTION = false;
private static String PAYPAL_ACCESS_TOKEN;
private static void initPayPal() {
InputStream is = null;
try {
is = ApplicationConfig.class.getResourceAsStream(
IS_PRODUCTION? "/my_paypal_sdk_config.properties" : "/my_paypal_sdk_config_test.properties");
PayPalResource.initConfig(is);
String clientID = ConfigManager.getInstance().getConfigurationMap().get("clientID");
String clientSecret = ConfigManager.getInstance().getConfigurationMap().get("clientSecret");
PAYPAL_ACCESS_TOKEN = new OAuthTokenCredential(clientID, clientSecret).getAccessToken();
} catch (Exception e) {
throw new RuntimeException(e);
} finally {
IOUtils.closeQuietly(is);
}
}
and while trying to get the clientID I have
java.io.IOException: Resource 'sdk_config.properties' could not be found
Strange behavior - I thought I've just configured the sdk to use my own properties file.
Please advice how could I set up those settings properly!
So here is the solution I found:
Create an empty sdk_config.properties file in default location
Load your own properties:
private static void initPayPal() {
InputStream is = null;
try {
is = ApplicationConfig.class.getResourceAsStream(
IS_PRODUCTION ? "/my_paypal_sdk_config.properties" : "/my_paypal_sdk_config_test.properties");
Properties props = new Properties();
props.load(is);
PayPalResource.initConfig(props);
ConfigManager.getInstance().load(props);
String clientID = ConfigManager.getInstance().getConfigurationMap().get("clientID");
String clientSecret = ConfigManager.getInstance().getConfigurationMap().get("clientSecret");
PAYPAL_ACCESS_TOKEN = new OAuthTokenCredential(clientID, clientSecret).getAccessToken();
} catch (Exception e) {
throw new RuntimeException(e);
} finally {
IOUtils.closeQuietly(is);
}
}
We have made some good improvements to the PayPal Java SDK on integration steps. We are removing the need for sdk_config.properties file as they do not work as well, specially for multi-configuration settings.
Now, all you do is create an APIContext instance with clientId, clientSecret, and mode. You pass that context object for any API operation from there on.
Here is how the code would look like for different configurations:
APIContext defaultContext = new APIContext(clientId1, clientSecret1, "sandbox");
APIContext sandboxContext = new APIContext(clientId2, clientSecret2, "sandbox");
APIContext someOtherContext = new APIContext(clientId3, clientSecret3, "live");
APIContext liveContext = new APIContext(clientId, clientSecret, "live");
// Now pass any of the above context in these calls, and it would use those configurations.
Payment payment = new Payment();
// Fill in all the details.
payment.create(defaultContext);
// Replace that defaultContext with any of those other contexts.
Here is the wiki page explaining that: https://github.com/paypal/PayPal-Java-SDK/wiki/Making-First-Call
I had the same error with SDK 0.11 version. I use my own properties file, but code still looked for "sdk_config.properties". I put it into root in my CLASSPATH, but still got the same error. Then I made obvious and horrible solution: put empty "sdk_config.properties" into "rest-api-sdk-0.11.0.jar" library. This street magic solved my problem.