We Keep Coding

How to fix resource changed on src filesystem issue

I'm trying to use Hive on MR executing SQL and it fails half way with errors below:
Application application_1570514228864_0001 failed 2 times due to AM Container for appattempt_1570514228864_0001_000002 exited with exitCode: -1000
Failing this attempt.Diagnostics: [2019-10-08 13:57:49.272]Failed to download resource { { s3a://tpcds/tmp/hadoop-yarn/staging/root/.staging/job_1570514228864_0001/libjars, 1570514262820, FILE, null },pending,[(container_1570514228864_0001_02_000001)],1132444167207544,DOWNLOADING} java.io.IOException: Resource s3a://tpcds/tmp/hadoop-yarn/staging/root/.staging/job_1570514228864_0001/libjars changed on src filesystem (expected 1570514262820, was 1570514269265
The key message from the error log from my perspective is libjars changed on src filesystem (expected 1570514262820, was 1570514269265. There are several threads about this issue at SO but not been answered yet, like thread1 and thread2.
I found something valuable from apache jira and redhat bugzilla. I synced clock by NTP through all nodes related. But same issue is still there.
Any comment is welcomed, thx.

I still didn't know why the timestamp of resource file is inconsistent and there isn't a way to fix it in configuration way, AFAIK.
However, I managed to find a workaround to skip the issue. Let me summarize it here for anyone who might run into same issue.
By checking error log and search it at Hadoop source code, we can trace the issue at hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java.
Just remove the exception throwing statements,
private void verifyAndCopy(Path destination)
throws IOException, YarnException {
final Path sCopy;
try {
sCopy = resource.getResource().toPath();
} catch (URISyntaxException e) {
throw new IOException("Invalid resource", e);
}
FileSystem sourceFs = sCopy.getFileSystem(conf);
FileStatus sStat = sourceFs.getFileStatus(sCopy);
if (sStat.getModificationTime() != resource.getTimestamp()) {
/**
throw new IOException("Resource " + sCopy +
" changed on src filesystem (expected " + resource.getTimestamp() +
", was " + sStat.getModificationTime());
**/
LOG.debug("[Gearon][Info] The timestamp is not consistent among resource files.\n" +
"Stop throwing exception . It doesn't affect other modules. ");
}
if (resource.getVisibility() == LocalResourceVisibility.PUBLIC) {
if (!isPublic(sourceFs, sCopy, sStat, statCache)) {
throw new IOException("Resource " + sCopy +
" is not publicly accessible and as such cannot be part of the" +
" public cache.");
}
}
downloadAndUnpack(sCopy, destination);
}
Build hadoop-yarn-project and copy 'hadoop-yarn-common-x.x.x.jarto$HADOOP_HOME/share/hadoop/yarn`.
Leave this thread here and thanks for any further explanation about how to fix it without changing hadoop source.

I had to do the same , this should be configurable, even small latency will fail the execution, this might happen, if one changes the hadoop file system to use s3 and run MR program , Note* please make sure, you are using same jdk version to generate the jar as mentioned in apache hadoop docs for your hadoop version, else you might run into errors.

JavaFX WebEngine - fail to load firebug-lite.js that is stored locally

I'm trying to make kind of wrapper class to create applications that use JavaFX WebView for their GUI (I called the class Application), that I can easily use in my other apps (I make a jar of my wrapper project, and then jar can be imported in any other project). I assumed that including some debug features in my wrapper is important, so I downloaded 'firebug-lite.js' and 'firebug-lite.css' version 1.2 to ./src/firebug/ directory in my project folder (so no internet required). After a number of tries I came to following code in Application.debug():
public void debug() {
// file is inside the jar, in ./firebug/ directory, so I use getResourceAsStream()
BufferedReader br = new BufferedReader(new InputStreamReader(
getClass().getResourceAsStream("/firebug/firebug-lite.js")));
String line;
String wholeFirebugCode = "";
try {
// reading firebug-lite.js line-by-line, not including \n characters
while ((line = br.readLine()) != null) {
wholeFirebugCode += line;
}
if (wholeFirebugCode.isEmpty()) System.out.println("!");
else {
// everything is OK, file is read, so add firebug to our page
engine.executeScript("document.body.innerHTML += \""+
"<script type='text/javascript'>" +
wholeFirebugCode +
"</script>\"");
}
} catch (IOException e) {
printError(e, "during load of /firebug/firebug-lite.js from within jar (WebEngineApp.jar)");
}
}
So, I built a jar, checked whether it has ./firebug/firebug-lite.js (it does), included that jar in a sample project and tried with this code:
// constructor works fine
Application app = new Application("title of window", path_to_sample_html_page,
width, height, bridge);
app.run();
try {//we wait a few til page is fully loaded (just for sure)
TimeUnit.SECONDS.sleep(5);
} catch (InterruptedException e) {
printError(e, "");
}
// here I check if java-js connection is established and page is loaded. It is
Platform.runLater(app::test);
// here I run the code which cause problem
Platform.runLater(app::debug);
But this doesn't work. I use Java 8, if it makes any difference:
Exception in thread "JavaFX Application Thread" netscape.javascript.JSException: SyntaxError: Unexpected identifier 'css'
at com.sun.webkit.dom.JSObject.fwkMakeException(JSObject.java:146)
at com.sun.webkit.WebPage.twkExecuteScript(Native Method)
at com.sun.webkit.WebPage.executeScript(WebPage.java:1509)
at javafx.scene.web.WebEngine.executeScript(WebEngine.java:1005)
at emeshka.webengineapp.Application.debug(Application.java:157)
at Main.lambda$main$0(Main.java:41)
(... etc)
File seems to be read correctly, I've checked it by logging. Same thing happens if I try to use 'firebug-lite-compressed.js', and if I use this method of code including:
engine.executeScript(wholeFirebugCode);
I'm completely lost. It couldn't be that firebug is syntactically incorrect. Sample html page contains no js. How can I make firebug work?
I've tried to use these solutions, that require internet, but they just took no effect, and no errors were shown:
Html/Javascript debugging in JavaFX WebView
,
JAVAFX / WebView / WebEngine FireBugLite or Some other debugger?

Manually loading native libraries to circumvent a restrictive environment

I'm maintaining a Java Swing application that requires a connection to an instance of Microsoft SQL Server. For various reasons, I opted to replace the native SQL Server driver being used with jTDS (the aforementioned Microsoft drivers were not working at the time and have apparently failed in the field as well). When I try to run the executable .jar outside of the IDE, I run into issues because I'm missing the appropriate ntlmauth.dll dependency.
Before proceeding, it's important to note that this application is being developed and used in an extremely restrictive (Windows-only) environment:
I cannot install any software that requires Windows UAC authentication
My users cannot install or run any software that requires UAC authentication
This currently means I cannot write files to System32 or JAVA_HOME, and cannot use any sort of ProcessBuilder tomfoolery to start another JVM with whatever command line arguments I need
I cannot use executable wrappers/installers that would only require the UAC permission for the first time installation/setup
The solution I'm trying is a combination of this one and this one to check it--essentially packaging the .dll inside of the .jar, then extracting it and loading it if necessary--as most of the other solutions I've found have been incompatible with the above restrictions; however, I'm running into an issue where even after the native library is ostensibly "loaded," I get an exception saying it isn't.
My pre-startup code:
private static final String LIB_BIN = "/lib-bin/";
private static final String JTDS_AUTH = "ntlmauth";
// load required JTDS binaries
static {
logger.info("Attempting to load library {}.dll", JTDS_AUTH);
try {
System.loadLibrary(JTDS_AUTH);
} catch (UnsatisfiedLinkError e) {
loadFromJar();
}
try {
// do some quick checks to make sure that went ok
NativeLibraries nl = new NativeLibraries();
logger.debug("Loaded libraries: {}", nl.getLoadedLibraries().toString());
} catch (NoSuchFieldException ex) {
logger.info("Native library checker load failed", ex);
}
}
/**
* When packaged into JAR extracts DLLs, places these into
*/
private static void loadFromJar() {
// we need to put DLL in temp dir
String path = ***;
loadLib(path, JTDS_AUTH);
}
/**
* Puts library to temp dir and loads to memory
*/
private static void loadLib(String path, String name) {
name = name + ".dll";
try {
// have to use a stream
InputStream in = net.sourceforge.jtds.jdbc.JtdsConnection.class.getResourceAsStream(LIB_BIN + name);
// always write to different location
File fileOut = new File(System.getProperty("java.io.tmpdir") + "/" + path + LIB_BIN + name);
logger.info("Writing dll to: " + fileOut.getAbsolutePath());
OutputStream out = FileUtils.openOutputStream(fileOut);
IOUtils.copy(in, out);
in.close();
out.close();
System.load(fileOut.toString());
} catch (Exception e) {
logger.error("Exception with native library loader", e);
JOptionPane.showMessageDialog(null, "Exception loading native libraries: " + e.getLocalizedMessage(), "Exception", JOptionPane.ERROR_MESSAGE);
}
}
As you can see, I basically copied the solution from the first link verbatim, with a few minor modifications just to try and get the application running. I also copied the class from the second link and named it NativeLibraries, the invocation of that method is fairly irrelevant but it shows up in the logs.
Anyway here are the relevant bits of the log output on starting up the application:
2015-07-20 12:32:33 INFO - Attempting to load library ntlmauth.dll
2015-07-20 12:32:33 INFO - Writing dll to: C:\Users\***\lib-bin\ntlmauth.dll
2015-07-20 12:32:33 DEBUG - Loaded libraries: [C:\Program Files\Java\jre1.8.0_45\bin\zip.dll, C:\Program Files\Java\jre1.8.0_45\bin\prism_d3d.dll, C:\Program Files\Java\jre1.8.0_45\bin\prism_sw.dll, C:\Program Files\Java\jre1.8.0_45\bin\msvcr100.dll, C:\Program Files\Java\jre1.8.0_45\bin\glass.dll, C:\Program Files\Java\jre1.8.0_45\bin\net.dll, C:\Users\***\lib-bin\ntlmauth.dll]
2015-07-20 12:32:33 INFO - Application startup
***
2015-07-20 12:32:36 ERROR - Database exception
java.sql.SQLException: I/O Error: SSO Failed: Native SSPI library not loaded. Check the java.library.path system property.
at net.sourceforge.jtds.jdbc.TdsCore.login(TdsCore.java:654) ~[jtds-1.3.1.jar:1.3.1]
at net.sourceforge.jtds.jdbc.JtdsConnection.<init>(JtdsConnection.java:371) ~[jtds-1.3.1.jar:1.3.1]
at net.sourceforge.jtds.jdbc.Driver.connect(Driver.java:184) ~[jtds-1.3.1.jar:1.3.1]
at java.sql.DriverManager.getConnection(Unknown Source) ~[na:1.8.0_45]
at java.sql.DriverManager.getConnection(Unknown Source) ~[na:1.8.0_45]
One can see that the library was, indeed, "loaded," from the third line in the log (it's the last entry, if you don't feel like scrolling). However, I simply used the class that I felt like was probably using the native libraries (I also tried the TdsCore class to no avail), as the example that showed how to do this was just using a random class from the package the library was needed in.
Is there something I'm missing here? I'm not very experienced with the JNI or the inner workings of ClassLoaders, so I might just be loading it wrong. Any advice or suggestions would be greatly appreciated!

Welp I figured out a workaround: I ended up using JarClassLoader. This basically entailed copying all my dependencies, both Java and native, into a "libraries" folder within my main .jar, and disabling .jar signing in the IDE. The application is then run by a new class that simply creates a new JarClassLoader object and running the "invokeMain" method--an example is on the website. The whole thing took about three minutes, after several days of banging my head against a wall.
Hope this helps someone someday!

java Jtextfiled as google search bar?

Anyone knows how build a Google toolbar to search on web with java?
I have this code but something is wrong.
private void jTextField1ActionPerformed(java.awt.event.ActionEvent evt)
{
try {
String searchText = URLEncoder.encode(jTextField1.getText(),"UTF-8");
URLDisplayer.getDefault().showURL
(
new URL
(
"http://www.google.com/search?hl=en&q="+searchText+"&btnG=Google+Search"
)
);
} catch (Exception eee) {
return;
}

What I did:
called www.google.com and used the key words "java URLDisplayer". One of the result was: http://www.labath.org/docs/java/jdk1.2.2/netbeans/OpenAPIs/org/openide/awt/HtmlBrowser.URLDisplayer.html
I called www.google.com and used the key words "download org.openide.awt" (The package of the above class). One of the result was:
http://bits.netbeans.org/download/trunk/jnlp/org-openide-awt/ .
What you can do is following:
Download this jar file and add it to your class path from your Java project. Refresh and rebuild your project.
I tried it with my Eclipse installation, used your code and all compiler errors and warnings disappeared.

why does Files.probeContentType return null

i am using java se7 on mac, the oracle preview.
My problem is that "Files.probeContentType" returns null...is it possible that its due to the early status of se7 for mac?
My code:
if(directory == null) return;
String content = null;
try {
content = Files.probeContentType(directory.toPath());
} catch (IOException e) {
JOptionPane.showMessageDialog(main, e.toString());
return;
}
if(content == null)
{
return;
}
else if(content.contains("image"))
{
main.pctviewer.setImage(directory);
}
the name of the file is:
"/Users/admin/Desktop/temp/q12/formulare/Bildschirmfoto 2012-09-11 um 17.57.59.png"
and in debug mode in eclipse if i hover above File "file path = Unis-path(id:145)" is red

I have reported the bug to oracle again, hoping they will backport the jdk8 solution (I don't have much hope but you never know).
In the meantime you can use my own backport of the FileTypeDetector available at https://github.com/jeantil/jdk7-mimeutils the maven project packages to a jar which can be added to your classpath to enable mime type detection. I also provide a mime.types file to put in your home folder for the detection to work correctly. I extracted the mime.types file from some version of apache so it's pretty complete.

I found that the FileTypeDetector is buggy on OS X: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7133484
Apparently this will be fixed in Java 8.

Use the below approach to get mime type of file in Java 8 :
String location = "/Users/user/Desktop/leaf.jpg";
File file = new File(location);
Path source = Paths.get(location);
MimetypesFileTypeMap m = new MimetypesFileTypeMap(source.toString());
System.out.println( m.getContentType(file) );
Output of the above code is : 'image/jpg'