Right now, I am running into the following issue exactly. Specifically, spark-submit is attempting to connect to the yarn.resourcemanager at location 0.0.0.0/0.0.0.0.
I have checked all of the logs delineated in the stack overflow thread above. They all seem to be correct. I have also added in a yarn.resourcemanager.address=... line to the default settings files at the top of the spark configuration directories, exported YARN_CONF_DIR and all of the other fixes listed on that thread.
At the bottom of the comments in the top rated answer, a commentator pointed out that if none of the above fixes work, then spark is not using the correct configurations.
At this point, I am pretty sure that my spark install is not using the correct configurations (I did not install it).
How does one go about determining which configurations spark is using, and how does one change them to the correct configurations? (or maybe I just need to reboot the machine?)
In spark-shell for example, I can do this:
scala> getClass.getClassLoader.getResource("yarn-site.xml")
res1: java.net.URL = file:/etc/spark2/conf.cloudera.spark2_on_yarn/yarn-conf/yarn-site.xml
...and the result shows the exact resolved location of a config file from my current classpath. Same could be easily translated to Java (almost verbatim) if your application is Java-based.
You can try to access creationSite field on org.apache.spark.sql.SparkSession through debugger or through reflection. Then you can try to find class and place in code where your spark session is created and then you can try to find how org.apache.spark.sql.SparkSession.Builder is called.
Related
I am trying to get started with Cassandra triggers, but I cannot get Cassandra to load them. I have built jar files from here and here, and put them under C:\Program Files\DataStax-DDC\apache-cassandra\conf\triggers. I have restarted the DataStax_DDC_Server service (on Windows) and reopened the CQLSH command line, but trying to use the trigger class in a create trigger command gives me only:
ConfigurationException: <ErrorMessage code=2300 [Query invalid because of configuration issue] message="Trigger class 'org.apache.cassandra.triggers.InvertedIndex' doesn't exist">
I checked the jar files, and they include the class files.
The only thing I could find in the log files of cassandra is Trigger directory doesn't exist, please create it and try again. But I don't know if that is relevant.
EDIT: Following the last line shown here, I edited the cassandra.bat file. Now if I stop the DataStax_DDC_Server service and run the bat file directly, the create trigger command succeeds. Nevertheless, the service seems to be independent of this bat file. The question now is how to apply the same config to the service?
After googling creatively, I found a solution. As mentioned here you need to explicitly set the cassandra.triggers_dir variable, but for the service to pick it up, as explained here, you must configure it in the registry. So the answer is to update the registry key
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\DataStax_CDC_Server\Parameters\Java\Options
and add the line
-Dcassandra.triggers_dir=C:\Program Files\DataStax-DDC\apache-cassandra\conf\triggers
Note that the path should not be enclosed in quotations, or it won't work.
Don't forget to restart the service.
Above solution is working for window. it's difficult in window to find registry option. so to find registry option go to start menu and type "regedit" it will open registry window then you can do above settings.
I have to create some reports using LibreOffice. I need to connecto a Firebird database using JDBC. I installed the driver (adding it to the classpath) and configured my connection to use
firebirdsql://server:alias
org.firebirdsql.jdbc.FBDriver
when I test the class I get this error:
"could not load driver class org.firebirdsql.jdbc.FBDriver"
javax/resource/ResourceException
I tried in another machine and got the same message. Is there anything extra that should be added to classpath? What I´m missing here? I have both JRE and JDK installed and I´m using the first located in C:\Program Files(x86)\Java\jre7. There is no classpath environment variable defined nor global or for the user
The above continues to be correct for Jaybird 4.0. Here are instructions in a little more detail, based on Debian 10. This is for an existing Firebird database that you can access already (i.e. can connect to and have an established username and perhaps pw).
Start LibreOffice but not Base - use Calc or just the general LibreOffice icon. Go to Tools, Options, LibreOffice and click on Advanced (in the top section, not in the Base options). Check the version number of the installed JRE. Leave the Advanced window open.
Download the corresponding version of the Jaybird zip file and unpack it. As pointed out by Mark Rotteveel in the original answer, jaybird has a dependency which may mean the bare jar file won't work. The simplest way around that is to use the "full" jar file:
jaybird-full-4.0.0.java11.jar
You may wish to move it into /usr/share/java for consistency but will probably need root permissions to do so.
Back in LibreOffice, click on the Class Path button in the Advanced window, then on Add Archive and add the .jar file. Close the Advanced window and restart LibreOffice.
Connecting: start LO Base. The Database Connect Wizard will appear. Select
"Connect to an existing database". Specify JDBC as the driver and click Next.
Base supplies the jdbc: portion of the connect string, so add
firebirdsql:oo://host[:port]//<path to database file>
So, for example:
firebirdsql:oo://localhost:3050//home/dave/Firebird/employee.fdb
Note two occurances of "//" when using a full path.
When using an alias, there's only one set:
firebirdsql:oo://localhost:3050/employee
The JDBC driver class is:
org.firebirdsql.jdbc.FBDriver
Click Test Class to confirm that it is working. If not, retrace the steps above. The class test is independent of the connection string so no need to fill in both as you troubleshoot.
From that point on it should be self-evident. You'll end up with a .odb file that you can use to create Views, Reports, etc. on the industrial-strength Firebird back end.
Jaybird has a required dependency on JCA (Java Connector Architecture), specifically JCA 1.5 (although newer should work as well), as documented in the Jaybird 2.2.5 releasenotes,
This means you need to include either connector-api-1.5.jar (from the lib folder) on the classpath, or you need to use jaybird-full-2.2.5.jar as that includes the classes from connector-api-1.5.jar.
Also your url is incorrect on two things: The format of the url is
jdbc:firebirdsql[:<subprotocol>]://<host>[:<port>]/<path-or-alias>
Which mean that :alias is wrong (it should be /alias). And as also remarked by Gord Thompson in the comments, you need to use the subprotocol oo, so you need to use protocol prefix jdbc:firebirdsql:oo to address difference in interpretation on JDBC, as also documented here. So your url should probably look like:
jdbc:firebirdsql:oo://server/alias
I am working with the Vert.x Gradle template hosted at the Vert.x Github space.
The build file suggests that there is a runModIDEA target that runs IDEA-built class files so that rebuild/redeploy is not required to pick up changes:
runModIDEA - run the module from the project resources in IDEA. This allows you to run the module without building it
first!
... yet the task does not exist per ./gradlew tasks.
I am not tied to this particular build task per se.
I just want a working auto-redeploy solution that enables me to see updates without a two minute rebuild/redeploy cycle.
EDIT: I also tried running it directly, pointing to InteliJ IDEA output classpath. It works fine, but doesn't pick up changes.
vertx runmod com.mycompany~vert-x-reverse-proxy~1.0.0-final -c conf.json -cp out/production/vert-x-reverse-proxy
EDIT: I also tried ./gradlew runmod -m, first changing vertx_classpath.txt so that the IDEA files (out/production) are looked at first. Still no redeploy. In fact, while it was running, I deleted the out directory and it continued working.
EDIT: I also tried vertx run com.mycompany.myproject.ReverseProxyVerticle -c conf.json -cp out/production/vert-x-reverse-proxy... same results. It ran as expected but did not pick up changes. Only way to pick up changes was to gradlew clean and re-assemble.
EDIT: I have been through these instructions as well.
For anyone who stumbles upon this question, I had the same problem and managed to fix it by deleting everything under the /mods folder in the /target directory. This is in fact mentioned in the vertx documentation - though maybe could be a little more emphatic. Once everything under /mods is removed, start up the application and it redeploys whenever anything is changed.
If you are new to vertx and stumble with this problem or similar, it might be worth to have a look at this vertx google group entry. It describes the changes that need to be done to the generated project by the Vertx Gradle Template to get it running.
I know, this does not answer directly the question posted here but I hope it helps you further.
My problem is as follow : I am developing an eclipse plugin and sometimes I lose track of where the classes are loaded from. What I would like is to see in the debugger stacktrace where a class is loaded from (which jar on the harddrive).
Am I missing some obvious Eclipse option ?
Any help would be greatly appreciated !
To see location of classes loaded : In Run configurations -> Arguments tab add vm argument -verbose. That will include the full path of classes loaded.
EDIT:
To find a particular class from a list of jars:
I use a jar class finder plugin. I do not recall the exact location from where i downloaded, the alphaworks link seems broken as of now. But you can find similar tools.
Note : In eclipse, if the console fills up to a point you are not able to see previous logs, then in Common tab -> Standard I/O check File option.
I am working on a team project in Java. One requirement is that we dynamically populate a drop-down menu of all classes that implement a certain interface. New classes can be added after compile time. To accomplish this we are using reflection.
Problem: All of the drop-down menus are blank on my system. I cannot for the life of me figure out why they are not populating. All other 5 team members have it working on their system.
Things I tired that didn't work:
1) Installing most recent eclipse (galileo) because rest team was using it
2) Re-install most recent java release (jdk1.6.0-17 and jre6)
3) Check PATH and JAVA_HOME variables
Any thoughts as to what else I can try or if something I did should have solved it and didn't? It is driving me crazy.
Edit:
I should have been clearer that we are developing in a team. We are using SVN for version control and we are all running the exact same source code. I even tried checking out a fresh copy of the entire tree from SVN, but I had the same issue with reflection on my system while it worked for teammates.
The team created an executable jar and that ran on everyone's system fine except for mine. Everything worked for me except the reflection bit.
You need to debug your application. This means you have to systematically explore possible causes of the problem. Here are some things that come to mind:
Could your GUI be failing rather than reflection? What if you output with System.out.println() rather than your menu?
Is your reflection code throwing an exception, and are you ignoring it?
Is your reflection code actually being called? Toss a println() in there to be sure!
Is the test for the interface suffering from a typo or similar error that's causing it to fail? Try finding classes that implement Serializable instead!
Is your reflection test running in the main thread and trying to update your GUI? You need to use SwingUtilities.invokeAndWait to get an update to the Swing worker thread.
You're working with Eclipse; Eclipse has a fantastic debugger. Set a breakpoint near where your main action is and then single step through the code.
PATH and JAVA_HOME won't help. PATH only affects dynamically-linked libraries ("native code"). JAVA_HOME is a scripting variable that happens to be used by some Java-based utilities like Ant and Tomcat; it means nothing to the Java runtime itself.
You need to be investigating the classpath, which should be specified by the -classpath option to the java command, in the Build Path in your Eclipse project properties, or in the Class-Path attribute of the main section of a JAR file if you're launching java with the -jar option.
From within your code, you should be able to list the contents of your classpath by examining the system property, "java.class.path"
System.out.println(System.getProperty("java.class.path"));
Problem solution:
Classpath leading to source code must have no spaces in it.
I am running windows XP and, for whatever reason, if the classpath that leads to the jar file or source code that is using reflection has any spaces in it, then the reflection fails.
I took the jar file that works for the rest of my team and ran it from C:\ on my system and the reflection worked perfectly fine.
I do not know why this is so please comment if you know what is happening.
Might be a long shot, but look for differences in security settings for you and your team mates. Article describing more details http://www.ibm.com/developerworks/library/j-dyn0603/ heading "Security and reflection"