I am trying to poll the ftp location.
I'm using Jenkins for continuous integration of the projects.So, it would be helpful if anyone can suggest me with a plugin in Jenkins or any other method to watch over the changes in FTP location.
I need to monitor the changes in FTP location and as the changes are found I have to build another job.
Not sure how to do it with Jenkins, but if you want to monitor an FTP location for changes (i.e. receive notifications when files added/removed/modified in a directory) using plain java, then the following library can help you with the actual polling/notification mechanism: https://github.com/drapostolos/rdp4j (Remote Directory Poller for Java).
Simple usage example of the API:
package example
import java.util.concurrent.TimeUnit;
import com.github.drapostolos.rdp4j.DirectoryPoller;
import com.github.drapostolos.rdp4j.spi.PolledDirectory;
public class FtpExample {
public static void main(String[] args) throws Exception {
String host = "ftp.mozilla.org";
String workingDirectory = "pub/addons";
String username = "anonymous";
String password = "anonymous";
PolledDirectory polledDirectory = new FtpDirectory(host, workingDirectory, username, password);
DirectoryPoller dp = DirectoryPoller.newBuilder()
.addPolledDirectory(polledDirectory)
.addListener(new MyListener())
.setPollingInterval(10, TimeUnit.MINUTES)
.start();
TimeUnit.HOURS.sleep(2);
dp.stop();
}
}
The RDP4J User Guide:
provides an example of FtpDirectory class which lists files in an FTP location using appache commons FTPClient
describes what events MyListener can listen for
How to configure the DirectoryPoller
Not sure how you can achieve this in Jenkins. If I were to just answer monitoring the FTP location part here is how you can do this.
Determine what programming language you want to use. (Java, .NET etc). Write code to
monitor the FTP server (assuming it is a specific remote directory you want to monitor)
and execute the job that needs to be executed. Both the monitoring and the executing the
job needs to be done in the programming language.
I am also assuming that you need a timer of some sort to do the monitoring, this can
also be done using a programming language such as Java.
Related
i am trying to trigger a java function each time my IoT Hub receives a batch of 64 messages (or whatever, the number is not important). I followed this guide to create the basic code, then i edited creating this function
public class Function {
#FunctionName("ProcessIotMessages")
public void processIotMessages(
#EventHubTrigger(name = "message",
eventHubName = "samples-workitems",
connection = "HUB-1544-DEV_events_IOTHUB") List<String> messages,
final ExecutionContext context) {...Function Logic...}
}
The connection parameter is the IoT Hub connection string in formatted as event hub compatible endpoint (e.g. Endpoint=sb://iothub-hostname-blablabla).
I package and deploy this code with maven plugins specified in the guide linked above. The deploys works fine, i can see mi function up and running with no errors from the portal, the HUB-1544-DEV_events_IOTHUB setting app is correctly created with the correct connection string.
The only strange thing i notice in the portal is in the trigger blade. As you can see, cardinality is One, while it should be set to many, since i did not specify the cardinality parameter in the function. The default one is many according to this guide. This makes me think that i not being able to pass the correct trigger syntax.
Anyway, the problem is that this function is not starting either from my local machine or the portal. Any suggestions? Thx
As #evilSnobu posted in the comments, the problem was the event hub name. Just go to Portal -> your IoT Hub -> Built-in endpoints and find all the information to configure the trigger in there.
Suppose you have Spark + Standalone cluster manager. You opened spark session with some configs and want to launch SomeSparkJob 40 times in parallel with different arguments.
Questions
How to set reties amount on job failures?
How to restart jobs programmatically on failure? This could be useful if jobs failure due lack of resources. Than I can launch one by one all jobs that require extra resources.
How to restart spark application on job failure? This could be useful if job lack resources even when it's launched simultaneously. Than to change cores, CPU etc configs I need to relaunch application in Standalone cluster manager.
My workarounds
1) I pretty sure the 1st point is possible, since it's possible at spark local mode. I just don't know how to do that in standalone mode.
2-3) It's possible to hand listener on spark context like spark.sparkContext().addSparkListener(new SparkListener() {. But seems SparkListener lacks failure callbacks.
Also there is a bunch of methods with very poor documentation. I've never used them, but perhaps they could help to solve my problem.
spark.sparkContext().dagScheduler().runJob();
spark.sparkContext().runJob()
spark.sparkContext().submitJob()
spark.sparkContext().taskScheduler().submitTasks();
spark.sparkContext().dagScheduler().handleJobCancellation();
spark.sparkContext().statusTracker()
You can use SparkLauncher and control the flow.
import org.apache.spark.launcher.SparkLauncher;
public class MyLauncher {
public static void main(String[] args) throws Exception {
Process spark = new SparkLauncher()
.setAppResource("/my/app.jar")
.setMainClass("my.spark.app.Main")
.setMaster("local")
.setConf(SparkLauncher.DRIVER_MEMORY, "2g")
.launch();
spark.waitFor();
}
}
See API for more details.
Since it creates process you can check the Process status and retry e.g. try following:
public boolean isAlive()
If Process is not live start again, see API for more details.
Hoping this gives high level idea of how we can achieve what you mentioned in your question. There could be more ways to do same thing but thought to share this approach.
Cheers !
check your spark.sql.broadcastTimeout and spark.broadcast.blockSize properties, try to increase them .
I am trying to develop a web server using Java's WebSockets. I am using Eclipse with Tomcat 7, so the javax.websocket-jar file is already included.
I created a simple ServerEndpoint
import javax.websocket.OnMessage;
import javax.websocket.server.ServerEndpoint;
#ServerEndpoint("/login")
public class LoginServer {
#OnMessage
public String onMessage(String message) {
User foobar = new User("Foo", "Bar", "Baz");
return foobar.toString();
}
}
Then I opened the console in Firefox and tried to connect:
var ws = new WebSocket('ws://localhost:8080/MyServer/login');
This ends in an error message which says it can't connect (404).
So I searched the web and found this Jetty thing. I don't really want to add an unnecessary layer on top of my software, but I couldn't find a way to connect to my server.
Do I need to use Jetty when programming WebSockets in Eclipse? If so, why?
I fixed it. I wanted to have a minimal example so I wanted to add the methods bit by bit. But I needed the #OnOpen method. Now it works. (Embarrassing...)
I have a requirement to add group members to an IBM Domino Group through java code. I am using Notes.jar to connect to IBM Domino v9.0, and my java code is running on a different machine, then the Domino machine.
From the Domino documentation I found out that "AdministrationProcess" class needs to be used to add member to group. But when i am trying to create "AdministrationProcess" object by calling session.createAdministrationProcess("IBMDominoServer"). I am getting the error Restricted operation on a server.
My test code is as follows
public class LotusDomino{
public static void main(String args[]) throws Exception{
String[] argv = {"192.168.2.111","Administrator","<password>"};
deleteUser(argv[0], argv[1], argv[2]);
}
private static void deleteUser(String host, String userName, String password) throws Exception{
Session s = NotesFactory.createSession(host, userName, password);
try{
AdministrationProcess process = s.createAdministrationProcess("IBMDominoServer.xanadufinancials.com");
}catch(NotesException e){
System.err.println("exception --- "+e.id+":"+e.text+":"+e.internal); // this prints the following error : exception --- 4183:Restricted operation on a server:null
}
}
The code shows same error irrespective of what i pass in as the server Name . So it shouldn't be a code issue. I did a little bit of search for this, and found out that Administrator should have editor access on admin4.nsf. Verified the access it was present.
Please let me know what can be the issue. Thanks in advance.
Using the Administration Process is one way to add a user to a group, and it is the safest way when you have no knowledge of how directory services on the Domino server have been configured. But in most basic configurations, adding a user to a group is very simple. You open the names.nsf database, open the Groups view, locate the document for the group, and add the name to the list stored in the Members item. The one catch is that if the Members list is too long, you may have to write code that is capable of divide it into subgroups (and/or code to detect the pattern of existing subgroups and add to them instead).
Regarding using the NotesAdministrationProcess class, if we can trust that the error message means what it says it means, then your problem is that the user id you are using does not have permission to run restricted operations on the server. Here is a link to info about server configuration for agent permissions. If you're using NCSO.jar (see my question above), then be a separate configuration for users permitted to perform restricted operations over IIOP, but I'm not sure and my server is down at the moment so I can't check.
I need to use HDFS cluster from remote desktop through Java API. Everything works OK until it comes to write access. If I'm trying to create any file I receive access permission exception. Path looks good but exception indicates my remote desktop user name which is of course is not what I need to access needed HDFS directory.
The question is:
- Is there any way to represent different user name using 'simple' authentication in Java API?
- Could you please point some good explanation of authentication / authorization schemes in hadoop / HDFS preferable with Java API examples?
Yes, I already know 'whoami' could be overloaded in this case using shell alias but I prefer to avoid solutions like this. Also specifics here is I dislike usage of some tricks like pipes through SSH and scripts. I'd like to perform everything using just Java API.
Thank you in advance.
After some studying I came to the following solution:
I don't actually need the full Kerberos solution, it is enough currently that clients can run HDFS requests from any user. Environment itself is considered secure.
This gives me solution based on hadoop UserGroupInformation class. In future I can extend it to support Kerberos.
Sample code probably useful for people both for 'fake authentication' and remote HDFS access:
package org.myorg;
import java.security.PrivilegedExceptionAction;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FileStatus;
public class HdfsTest {
public static void main(String args[]) {
try {
UserGroupInformation ugi
= UserGroupInformation.createRemoteUser("hbase");
ugi.doAs(new PrivilegedExceptionAction<Void>() {
public Void run() throws Exception {
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://1.2.3.4:8020/user/hbase");
conf.set("hadoop.job.ugi", "hbase");
FileSystem fs = FileSystem.get(conf);
fs.createNewFile(new Path("/user/hbase/test"));
FileStatus[] status = fs.listStatus(new Path("/user/hbase"));
for(int i=0;i<status.length;i++){
System.out.println(status[i].getPath());
}
return null;
}
});
} catch (Exception e) {
e.printStackTrace();
}
}
}
Useful reference for those who have a similar problem:
Cloudera blog post "Authorization and Authentication In Hadoop". Short, focused on simple explanation of hadoop security approaches. No information specific to Java API solution but good for basic understanding of the problem.
UPDATE:
Alternative for those who uses command line hdfs or hadoop utility without local user needed:
HADOOP_USER_NAME=hdfs hdfs fs -put /root/MyHadoop/file1.txt /
What you actually do is you read local file in accordance to your local permissions but when placing file on HDFS you are authenticated like user hdfs.
This has pretty similar properties to API code illustrated:
You don't need sudo.
You don't need actually appropriate local user 'hdfs'.
You don't need to copy anything or change permissions because of previous points.