I am currently setting up a Continuous Integration system with Jenkins, and I came across a problem :
Almost every project depends on others projects. So, in order to perform daily builds, I use the CloudBees Build Flow plugin. It does its job pretty nicely actually, but not in an optimal way : It builds EVERY jobs I tell it to, without even checking on Git if there are any changes. So I would like to know if there are any ways to force Jenkins to check on Git if there are any changes before actually building the project.
PS : Sorry for my English, I am not a native speaker
Not sure, if you have looked at the configs in the job settings. There is a place to force a fresh checkout. I have svn linked, similar thing will be with git
If not you can looking for adding manual commands as shown below. Check to see if you can arrange the order of this to execute first then build your task
In the end, I chose to stick to BuildFlow and the Groovy language, instead of using scripts, but it's just by convenience, and this solution would totally work with shell language. Moreover, using BuildFlow allows you to use Parallel(), to start multiple jobs at the same time.
Here's my solution :
I found the plugin Jenkins Poll SCM, that polls the SCM before trying to build it (only if necessary).
The only problem with CloudBees Build Flow plugin is that it does not wait for previous jobs to be completed, as I am not using the build() method. To overcome this problem, I made my own buildWithPolling() method, that waits for the job to be done before going on. The only downside of my method is that it does not wait for downstream jobs to be finished (But I don't know if it does with the build() method either...). Here is the code of my method :
def buildWithPolling(project)
{
//Connection to the URL starting the polling, and starting the building if needed
def address = "http://localhost:8080/jenkins/job/" + project + "/poll"
println "Connexion à " + address + " pour scrutation du Git et build si besoin est."
def poll = new URL(address)
poll.openStream()
//Declaration of variables used to know if the build is still running, or if it is finished
boolean inProgress = true
def parser = new XmlParser()
def rootNode = null;
address = "http://localhost:8080/jenkins/job/" + project + "/lastBuild/api/xml?depth=1&tree=building&xpath=*/building"
while(inProgress) {
//A 5 seconds pause, because we don't need to overload the server
sleep(5000)
//Request sent to the server, to know if the job is finished.
def baos =new ByteArrayOutputStream()
baos << new URL(address).openStream()
rootNode = parser.parseText(new String(baos.toByteArray()))
inProgress = rootNode.text().toBoolean()
}
}
It is probaly not the best solution, but it's working for me !
Related
I am extremely new the Android app development and Stack Overflow. I am trying to recreate traceroute in an Android app since Android devices do not come with traceroute by default. I've encountered a couple stack overflow posts talking about solutions to this, but I have still run into challenges.
Traceroute on android - the top post on this thread links an Android Studio project that implements traceroute using ping. If I understand the algorithm correctly, it continually pings the destination IP, incrementing the time-to-live field to obtain information about intermediary routers. I've tried to recreate this behavior, but for certain values of TTL, the ping stalls and doesn't retrieve any router information. I'm not really sure why this happens. Here's a quick demo function I spun up... at some point in the loop the pings stall.
public static void smallTracerouteDemoShowingThatTheProgramStallsAtCertainTTLs() {
try {
String host = "google.com";
int maxTTL = 20;
for (int i = 1; i < maxTTL; i++) {
// Create a process that executes the ping command
Process p = Runtime.getRuntime().exec("ping -c 1 -t " + i + " " + host);
// Get a buffered reader with the information returned by the ping
BufferedReader br = new BufferedReader(new InputStreamReader(p.getInputStream()));
// Convert the BufferedReader to a string
String dataReturnedByPing = "";
for (String line; (line = br.readLine()) != null; dataReturnedByPing += "\n"+line);
// Print out information about each TTL
System.out.println("TTL = " + i + " out of " + maxTTL);
System.out.println(dataReturnedByPing);
System.out.println("========================================");
}
} catch (Exception e) {
e.printStackTrace();
}
}
how to run traceroute command through your application? - The solution on this thread suggests using BusyBox. I've not used BusyBox as yet, but it seems like I would have to embed BusyBox into my app to get things to work. After doing some research it looks like BusyBox provides numerous Linux commands through one executable. I'm a bit hesitant to explore this option because I really only need the traceroute command. In addition, I know that Android targets a few different CPU architectures, and I'm not sure if one executable will support them all.
I've also run into a github repository that takes another approach to running traceroute:
https://github.com/wangjing53406/traceroute-for-android - In this repository the author embeds the traceroute source code into the project and uses the NDK to build the source code along with the rest of his app. I really like this approach because it feels the most "correct." It uses a built traceroute instead of a Java-based implementation, so you can't find yourself in a situation where the Java implementation gives you one thing and the actual traceroute gives you another. When I open this project to experiment with it, my build fails. The top line says:
org.gradle.initialization.ReportedException: org.gradle.internal.exceptions.LocationAwareException: A problem occurred configuring root project 'traceroute-for-android-master'.
Any help on why this happens or ways to troubleshoot it would be fantastic.
For reference, the minimum SDK I am targeting is API 21 and I am running on Android Studio 3.3.0.
So, at this point I'm stumped. If you were trying to make an app that would let you execute traceroute commands, how would you do it? I really like the NDK approach because it guarantees you're getting true traceroute behavior. If you have any guides to getting that set up for my Android version/SDK, I would appreciate if you would post them. If you'd take another approach I'd to hear about it as well.
Thank you in advance.
Am using StreamExecutionEnvironment and calling client = env.executeAsync() to execute my jobs.
Positive scenario is working fine. Am trying below negative scenario with save point,
Whenever user press cntl+c or force closes the app, am triggering save point like below(in my shutdown hook i do this),
client.stopWithSavepoint(false,"file:///flink/savepoints")
It is creating meta file and 2 more files in the location.
But when i start my app again, it is not restoring these states. In flink configuration i have added following settings with that i hope it should restore but it is not
conf = new Configuration();
conf.set(ExecutionCheckpointingOptions.ENABLE_CHECKPOINTS_AFTER_TASKS_FINISH, true);
conf.set(CoreOptions.DEFAULT_PARALLELISM, props.getParallelism());
conf.setString(StateBackendOptions.STATE_BACKEND, "filesystem");
conf.setString(CheckpointingOptions.CHECKPOINTS_DIRECTORY, props.getCheckPointStorage());
conf.set(CheckpointingOptions.FS_SMALL_FILE_THRESHOLD, MemorySize.ZERO);
conf.setString(CheckpointingOptions.SAVEPOINT_DIRECTORY, props.getSavePointStorage());
// setting restart strategy and related configuration.
// "fixeddelay" restart strategy
conf.set(RestartStrategyOptions.RESTART_STRATEGY, props.getRestartStrategy());
y
conf.set(RestartStrategyOptions.RESTART_STRATEGY_FIXED_DELAY_ATTEMPTS, props.getRestartAttempts());
conf.set(RestartStrategyOptions.RESTART_STRATEGY_FIXED_DELAY_DELAY,
Duration.ofSeconds(props.getRestartDelay()));
env.enableCheckpointing(props.getCheckPointingInterval(), CheckpointingMode.EXACTLY_ONCE);
env.getCheckpointConfig()
.setExternalizedCheckpointCleanup(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
What am i missing here?
As i have multi module java project, i am currently not sure how flink app is started. I understand that Save points can be restored easily by passing -s and savepointdirpath as arguments but how to do that in java programatically.
Thanks in advance.
I have run an example hadoop job, and when I look at the Jobs area of the Hue web app I can see the details for my job. I would like to access this info programmatically... I wrote the following code as a test:
JobClient jobClient = new JobClient(new Configuration());
JobStatus[] jobStatuses = jobClient.getAllJobs();
System.out.println("Found " + jobStatuses.length + " job statuses.");
for(JobStatus jobStatus : jobStatuses) {
System.out.println(jobStatus.getJobID());
}
jobClient.close();
Output is: "Found 0 job statuses."
Other details - I testing this using the CDH4 standalone VM. I am using the conf files from /etc/hadoop/conf/conf.cloudera.yarn1 (using the /etc/hadoop/conf ones did not work).
The question here seems related but is unanswered as well...
What are some areas that I could investigate to sort this out?
Thanks!
After some additional research I determined that I was using the mr2 (yarn) compatible jars instead of the mr1 compatible jars. I changed my pom.xml appropriately and the problems magically went away.
I have some segment of code I want to profile on many different inputs (~1000) so it doesn't make sense to manually run each test and save the results. I'm using yourkit in combination with Eclipse to profile. Is there any way to create "new sessions" for profiling? I want to be able to separate each run so that would make the most sense.
You don't really need to create "sessions" for each test. Instead, you have to capture a snapshot of the profiling data at the end of each test, and clear the profiling data before running the next test.
Using the yourkit API, you can do so in a manner similar to:
public void profile(String host, int port, List<InputData> inputDataSet) {
Map<InputData, String> pathMap = new HashMap<InputData, String>(); //If you want to save the location of each file
//Init profiling data collection
com.yourkit.api.Controller controller = new Controller(host, port);
controller.startCPUSampling(/*with your settings*/);
controller.startAllocationRecording(/*With your settings*/);
//controller.startXXX with whatever data you want to collect
for (InputData input: inputDataSet) {
//Run your test
runTest(inputData);
//Save profiling data
String path = controller.captureSnapshot(/*With or without memory dump*/);
pathMap.put(input, path);
//Clear yourkit profiling data
controller.clearAllocationData();
controller.clearCPUData();
//controller.clearXXX with whatever data you are collecting
}
}
I don't think you need to stop collecting, capture snapshot, clear data, restart collecting, you can just capture and clear data, but please double-check.
Once the tests are run, you can open the snapshots in yourkit and analyze the profiling data.
Unfotunately it's not clear how to run your tests. Does each test run in own JVM process or you run all tests in loop inside single JVM?
If you run each test in own JVM then you need 1) Run JVM with profiler agent, i.e. use -agentpath option (the details is here http://www.yourkit.com/docs/java/help/agent.jsp ). 2) Specify what you are profiling on JVM startup (agent option "sampling", "tracing", etc) 3) Capture snapshot file on JVM exit ("onexit" agent option).
Full list of options http://www.yourkit.com/docs/java/help/startup_options.jsp
If you run all tests inside single JVM you can use profiler API http://www.yourkit.com/docs/java/help/api.jsp to start profling before test starts and capture snapshot after test finishes. You need to use com.yourkit.api.Controller class.
I have multiple EBS-backed EC2 instances running and I want to be able to take a snapshot of the EBS volume behind one of them, create a new EBS volume from that snapshot, and then mount that new EBS volume onto another as an additional drive. I know how to do this via the AWS web console, but I would like to automate the process by using the AWS Java API.
If I simply call the following commands one after another:
CreateSnapshotResult snapRes
= ec2.createSnapshot(new CreateSnapshotRequest(oldVolumeID, "Test snapshot"));
Snapshot snap = snapRes.getSnapshot();
CreateVolumeResult volRes
= ec2.createVolume(new CreateVolumeRequest(snap.getSnapshotId(), aZone));
String newVolumeID = volRes.getVolume().getVolumeId();
AttachVolumeResult attachRes
= ec2.attachVolume(new AttachVolumeRequest(newVolumeID, instanceID, "xvdg"));
I get the following error:
Caught Exception: Snapshot 'snap-8e822cfd' is not 'completed'.
Reponse Status Code: 400
Error Code: IncorrectState
Request ID: 40bc6bad-43e0-49e6-a89a-0489744d24e6
To get around this, I obviously need to wait until the snapshot is completed before I create the new EBS volume from the snapshot. According to the Amazon docs, the possible values of Snapshot.getState() are "pending, completed, or error," so I decided to check in with AWS to see if the snapshot is still in the "pending" state. I wrote the following code, but it has not worked:
CreateSnapshotResult snapRes
= ec2.createSnapshot(new CreateSnapshotRequest(oldVolumeID, "Test snapshot"));
Snapshot snap = snapRes.getSnapshot();
System.out.println("Snapshot request sent.");
System.out.println("Waiting for snapshot to be created");
String snapState = snap.getState();
System.out.println("snapState is " + snapState);
// Wait for the snapshot to be created
while (snapState.equals("pending"))
{
Thread.sleep(1000);
System.out.print(".");
snapState = snapRes.getSnapshot().getState();
}
System.out.println("Done.");
When I run this, I get the following output:
Snapshot request sent.
Waiting for snapshot to be created
snapState is pending
.............................................
Where the dots continue to be printed until I kill the program. In the AWS Web Console, I can see that the snapshot has been created (it now has a green circle marking it as "completed"), but somehow my program has not gotten the message.
When I replace the while loop with a simple wait for a second (insert the line Thread.sleep(1000) after Snapshot snap = snapRes.getSnapshot(); in the first code snippet), the program will often create a new EBS volume without complaint (it then dies when I try to attach the volume to the new instance). Sometimes, however, I will get the IncorrectState error even after waiting for a second. I assume this means that there is some variance in the amount of time it takes to create a snapshot (even of the same EBS volume), and that one second is enough to account for some but not all of the possible delay times.
I could just increase the hard-coded delay to something sure to be longer than the expected time, but that approach has many faults (it waits unnecessarily for most of the times I will use it, it is still not guaranteed to be long enough, and it won't translate well into a solution for the second step, mounting the EBS volume onto the instance).
I would really like to be able to check in with AWS at regular intervals, check to see if the state of the snapshot has changed, and then proceed once it has. What am I doing wrong and how should I fix my code to allow my program to dynamically determine when the snapshot has been fully created?
EDIT: I've tried using getProgress() rather than getState() as per the suggestion. My changed code looks like this:
String snapProgress = snap.getProgress();
System.out.println("snapProgress is " + snapProgress);
// Wait for the snapshot to be created
while (!snapProgress.equals("100%"))
{
Thread.sleep(1000);
System.out.print(".");
snapProgress = snapRes.getSnapshot().getProgress();
}
System.out.println("Done.");
I get the same output as I did when using getState(). I think my problem is that the snapshot object that my code references is not being updated correctly. Is there a better way to refresh/update that object than simply calling its methods repeatedly? My suspicion is that I'm running up against some sort of issue with the way that the API handles requests.
Solved it. I think the problem was that the Snapshot.getState() call doesn't actually make a new call to AWS, but keeps returning the state of the object at the time it was created (which would always be pending).
I fixed the problem by using the describeSnapshots() method:
String snapState = snap.getState();
System.out.println("snapState is " + snapState);
System.out.print("Waiting for snapshot to be created");
// Wait for the snapshot to be created
while (snapState.equals("pending"))
{
Thread.sleep(500);
System.out.print(".");
DescribeSnapshotsResult describeSnapRes
= ec2.describeSnapshots(new DescribeSnapshotsRequest().withSnapshotIds(snap.getSnapshotId()));
snapState = describeSnapRes.getSnapshots().get(0).getState();
}
System.out.println("\nDone.");
This makes a proper call to AWS every time, and it works.
Instead of getstate() , try using getProgress() method. If you are getting it blank then your EBS snapshot is not ready. It gives output in string percentage format ( 100% when your snapshot is ready). Hopefully it should do the trick. Let me know if it works.