We Keep Coding

Spark submit in java(SparkLauncher)

I made spark+hadoop yarn enviroment and spark-submit command works well. So I made SparkLauncher java code to do this in my application jar, BUT somehow it doesn't work (actually computer fan is spinning at first but not as long as i did with spark-submit.)
It seems not work well (no application log in hadoop web ui, unlike spark-submit). I cannot see any error log when I do with 'SparkLauncher'. without log message, I can do nothing with it.
Here is how I made it so far.
public class Main {
public static void main(String[] args) {
Process spark = null;
try
{
spark = new SparkLauncher()
.setAppResource("/usr/local/spark/examples/jars/spark-examples*.jar")
.setMainClass("org.apache.spark.examples.SparkPi")
.setMaster("yarn")
.setDeployMode( "cluster")
.launch();
}
catch( IOException e)
{
e.printStackTrace();
}
}
}
executed it with ( java -jar example.jar)

I had the same problem at first. I think the main issue is that you forgot about the waitFor().
Also, it's really helpfull to extract your errorMessage and deal with it (e.g. log it or checking it while debuging ) within your java code. To allow this, you should create a streamReader thread as follows:
InputStreamReaderRunnable errorStreamReaderRunnable = new InputStreamReaderRunnable(spark.getErrorStream(), "error");
Thread errorThread = new Thread(errorStreamReaderRunnable, "LogStreamReader error");
errorThread.start();
int result= spark.waitFor();
if(result!=0) {
String errorMessage = extractExceptionMessage(errorStreamReaderRunnable.getMessage());
LOGGER.error(errorMessage);
}
This should be after your launch() command and inside your try block. Hope it helps

Method connect() from type PircBotX not visible

I am trying to create a bot using PircBotX, however I am unable to even get started making it. Using just the basic example code, I am unable to get the connect() method to work, it always give the error at compile mentioned in the title. Here is the code I am using:
import org.pircbotx.Configuration;
import org.pircbotx.PircBotX;
public class MyBot {
public static void main(String[] args) throws Exception {
Configuration configuration = new Configuration.Builder()
.setName("PircBotX") //Set the nick of the bot. CHANGE IN YOUR CODE
.setLogin("LQ") //login part of hostmask, eg name:login#host
.setAutoNickChange(true) //Automatically change nick when the current one is in use
.setCapEnabled(true) //Enable CAP features
.setServerHostname("irc.freenode.net")
.addAutoJoinChannel("#pircbotx") //Join the official #pircbotx channel
.buildConfiguration();
PircBotX bot = new PircBotX(configuration);
//Connect to server
try {
bot.connect();
} catch (Exception ex) {
ex.printStackTrace();
}
}
}
The error occurs with the line bot.connect();
PircBotX can be found at its Google Code page: https://code.google.com/p/pircbotx/

On the docs, there is a method called startBot(), should that be used instead?
void startBot() Start the bot by connecting to the server.
import org.pircbotx.Configuration;
import org.pircbotx.PircBotX;
public class MyBot {
public static void main(String[] args) throws Exception {
Configuration configuration = new Configuration.Builder()
.setName("PircBotX") //Set the nick of the bot. CHANGE IN YOUR CODE
.setLogin("LQ") //login part of hostmask, eg name:login#host
.setAutoNickChange(true) //Automatically change nick when the current one is in use
.setCapEnabled(true) //Enable CAP features
.setServerHostname("irc.freenode.net")
.addAutoJoinChannel("#pircbotx") //Join the official #pircbotx channel
.buildConfiguration();
PircBotX bot = new PircBotX(configuration);
//Connect to server
try {
//bot.connect();
bot.startBot();
} catch (Exception ex) {
ex.printStackTrace();
}
}
}

URLConnection created through OkHttp has useCache set to false

I am using the Kohsuke GitHub-API to connect to the GitHub from my Java (server-side) application and I wanted to use the OkHttp's ability to cache responses from the GitHub. This worked perfectly when I wrote a test for it, but it doesn't work in the application itself and I don't have a clue why that is. I have managed to trace the problem back to the creation of the URLConnection object that is created with its useCache variable set to false, but I cannot figure out why. Does it maybe have something to do with the server configuration or something like that?
I would appreciate any ideas or even a nudge in any direction, because frankly I don't have any ideas left... Thanks
Provider:
public class GitHubProvider implements Provider<GitHub> {
#Override
public GitHub get() {
GitHub gitHub = null;
HttpResponseCache cache = null;
OkHttpClient okHttpClient = new OkHttpClient();
File cacheDir = new File(System.getProperty("java.io.tmpdir"), UUID.randomUUID().toString());
try {
cache = new HttpResponseCache(cacheDir, 10L * 1024 * 1024);
} catch (IOException e) {
// NOTHING
}
okHttpClient.setResponseCache(cache);
try {
gitHub = GitHub.connectUsingPassword("user", "password");
} catch (Exception e) {
// NOTHING
}
gitHub.setConnector(new OkHttpConnector(okHttpClient));
return gitHub;
}
}
Test (works):
#RunWith(JukitoRunner.class)
public class SoftwareComponentServiceTest {
public static class Module extends TestModule {
#Override
protected void configureTest() {
bind(GitHub.class).toProvider(GitHubProvider.class);
}
}
#Inject
GitHub gitHub;
#Test
public void testInjectedGitHubResponseCache() {
try {
GHUser ghUser = gitHub.getUser("user");
GHRepository repository = ghUser.getRepository("repository");
int limit = gitHub.getRateLimit().remaining;
repository.getFileContent("README.md");
assertEquals(limit - 1, gitHub.getRateLimit().remaining);
repository.getFileContent("README.md");
assertEquals(limit - 1, gitHub.getRateLimit().remaining);
} catch (IOException e) {
e.printStackTrace();
}
}
}
Service that is used in the application (doesn't work):
#Singleton
#RequiresAuthentication
public class SoftwareComponentService {
#Inject
GitHub gitHub;
public List<SoftwareComponent> findAll() {
List<SoftwareComponent> softwareComponentList = new ArrayList<SoftwareComponent>();
try {
GHUser ghUser = gitHub.getUser("user");
List<GHRepository> repositories = ghUser.listRepositories().asList();
for (int i = 0; i < repositories.size(); i++) {
GHRepository repository = repositories.get(i);
if (!repository.getName().startsWith("sc_")) {
continue;
}
softwareComponentList.add(new SoftwareComponent(repository.getName(), repository.getDescription()));
}
} catch (IOException e) {
// NOTHING
}
return softwareComponentList;
}
}

The reason
The URLConnection object is created with its useCache variable set to false because its defaultUseCaches variable is also set to false by the Tomcat server at the time of initialization. Tomcat does this through its JreMemoryLeakPreventionListener class because reading resources from JAR files using java.net.URLConnections can sometimes result in the JAR file being locked (urlCacheProtection variable). The workaround they implemented to solve this problem was to disable URLConnection caching by default (!?!?).
The solution
The workaround to this workaround is to create a dummy URLConnection and use its setDefaultUseCaches() method to change the default value of every subsequently created URLConnection (as suggested by Jesse Wilson).
URL url = new URL("jar:file://dummy.jar!/");
URLConnection uConn = url.openConnection();
uConn.setDefaultUseCaches(true);
Big thanks to Jesse Wilson for pointing me in the right direction!

There's an insane method called URLConnection.setDefaultUseCaches() that could be doing it globally. That's an instance method that works like a static method: it sets the property for everyone.

How to use platformRequest to call a website Url?

I am making a j2me website launcher for an url "http://www.google.com".
I am using this code:
public class LuncherMidlet extends MIDlet {
boolean Isflage;
public void startApp() {
Isflage = false;
try {
Isflage = platformRequest("http://www.google.com");
} catch (ConnectionNotFoundException ex) {
ex.printStackTrace();
}
if(Isflage){
destroyApp(true);
notifyDestroyed();
}
}
public void pauseApp() {
}
public void destroyApp(boolean unconditional) {
}
}
This code is working fine, but when I try to close the browser of mobile it shows a message for connect to server. If I say no then it again open the this j2me launcher again. It's not exiting from mobile browser. My launcher reaches in a loop.
Kindly suggest me on this issue.

When you invoke platformRequest(), it would call to native apps, and on close on browser it is browser's stuff that is being invoked. You can't change it.

how to use Service,ServiceLocator,portType,porttypeProxy and stub to create a java client

i have used axis web service client wizard + develop client slider for generating files as :
Mage_Api_Model_Server_HandlerBindingStub
Mage_Api_Model_Server_HandlerPortType
Mage_Api_Model_Server_HandlerPortTypeProxy
MagentoService
MagentoServiceLocator
please look at my post
i am trying to make a client some thing like this:
package Magento;
public class MyClient {
public static void main(String[] args) {
try{
MagentoServiceLocator msl = new MagentoServiceLocator();
MagentoService ms = (MagentoService) msl.WHICH_METHOD_TO_CALL();
double product_list = ms.catalogProductList;
System.out.println("Product List: " + product_list);
} catch (Exception e) {
e.printStackTrace();
}
}
}
any help i have tried to the best of mine but not getting which method should i call for accessing a php webservice method.
any help?

Create a client for the given wsdl and try this:
WebServiceSoapBindingStub stub = new WebServiceSoapBindingStub();
stub._setProperty(Stub.ENDPOINT_ADDRESS_PROPERTY, wsdlUrl);
WebService service = (WebService) stub;
service.authenticateUser(username,password); //service.yourservicename
More information that will help you.
http://www.codeproject.com/KB/java/edujini_webservice_java.aspx