Iam Trying to import data from Mysql to hive using SqoopOptions but getting this error:
ERROR tool.ImportTool: Imported Failed: Wrong FS: hdfs://localhost:8020/user/hive/warehouse/default/emp/_logs, expected: file:///
it is importing data into hdfs but not in hive.
here is my complete code.
import org.apache.sqoop.tool.ImportTool;
import com.cloudera.sqoop.SqoopOptions;
public class App
{
public static void main( String[] args )
{
importToHive("emp");
}
/* CONSTANTS */
private static final String JOB_NAME = "Sqoop Hive Job";
private static final String MAPREDUCE_JOB = "Hive Map Reduce Job";
private static final String DBURL ="jdbc:mysql://localhost:3306/sample";
private static final String DRIVER = "com.mysql.jdbc.Driver";
private static final String USERNAME = "root";
private static final String PASSWORD = "cloudera";
private static final String HADOOP_HOME ="/usr/lib/hadoop-0.20-mapreduce";
private static final String JAR_OUTPUT_DIR = "/tmp/sqoop/compile";
private static final String HIVE_HOME = "/usr/lib/hive";
private static final String HIVE_DIR = "/user/hive/warehouse/";
private static final String WAREHOUSE_DIR = "hdfs://localhost:8020/user/hive/warehouse/default";
private static final String SUCCESS = "SUCCESS !!!";
private static final String FAIL = "FAIL !!!";
/* **
* Imports data from RDBMS MySQL and uploads into Hive environment
*/
public static void importToHive(String table){
System.out.println("SqoopOptions loading .....");
/* MySQL connection parameters */
SqoopOptions options = new SqoopOptions();
options.setConnectString(DBURL);
options.doOverwriteHiveTable();
options.setTableName(table);
options.setDriverClassName(DRIVER);
options.setUsername(USERNAME);
options.setPassword(PASSWORD);
options.setHadoopMapRedHome(HADOOP_HOME);
/* Hive connection parameters */
options.setHiveHome(HIVE_HOME);
options.setHiveImport(true);
options.setHiveTableName("bsefmcgh");
options.setOverwriteHiveTable(true);
options.setFailIfHiveTableExists(false);
//options.setFieldsTerminatedBy(',');
options.setOverwriteHiveTable(true);
options.setDirectMode(true);
options.setNumMappers(1); // No. of Mappers to be launched for the job
options.setWarehouseDir(WAREHOUSE_DIR);
options.setJobName(JOB_NAME);
options.setMapreduceJobName(MAPREDUCE_JOB);
options.setTableName(table);
options.setJarOutputDir(JAR_OUTPUT_DIR);
System.out.println("Import Tool running ....");
ImportTool it = new ImportTool();
int retVal = it.run((com.cloudera.sqoop.SqoopOptions) options);
}
}
I believe you do not need to specify the name node address in the warehouse-dir option.
Try this:
private static final String WAREHOUSE_DIR = "/user/hive/warehouse/default";
Related
i try to integrate cucumber scenarios in testrail, but i don't find some solutions, if somebody integrate it, can help?
i have scenarios write in java cucumber and need to integrate it in testrail. if it possible.
i try with api from documentation, but it is not work or i have some mistake
public static String TEST_RUN_ID = "5400";
public static String TESTRAIL_USERNAME = "username";
public static String TESTRAIL_PASSWORD = "password";
public static String RAILS_URL = "url";
public static final int TEST_CASE_PASSED_STATUS = 1;
public static final int TEST_CASE_FAILED_STATUS = 5;
public static void addResultForTestCase(String testCseId, int status,
String error) throws IOException, APIException {
String testRunId = TEST_RUN_ID;
APIClient client = new APIClient(RAILS_URL);
client.setUser(TESTRAIL_USERNAME);
client.setPassword(TESTRAIL_PASSWORD);
Map data = new HashMap();
data.put("status_id", status);
data.put("comment", "Test Executed - status updates automatically");
client.sendPost("add_result_for_case/" + testRunId+ "/" + testCseId + "", data);
}
}```
You might want to have a look at Railflow - it can parse Cucumber JSON reports and create/update tests and test runs in TestRail.
P.S. I'm developer of TestRail
I have created a Spring Boot microservice which runs aggregation on a stream of data and writes it into various Cassandra tables. I am looking for a java library similar to Flyway which will migrate Cassandra schema with the existence of a script in a resource folder. Does anyone have any recommendations, preferably for a library which you personally have used in production?
I used builtamont:
<dependency>
<groupId>com.builtamont</groupId>
<artifactId>cassandra-migration</artifactId>
<version>0.9</version>
</dependency>
migration in code:
import com.builtamont.cassandra.migration.CassandraMigration;
import com.builtamont.cassandra.migration.api.configuration.KeyspaceConfiguration;
import org.springframework.beans.factory.InitializingBean;
class CassandraDataSourceMigration implements InitializingBean {
private final String ip;
private final String clusterName;
private final Integer port;
private final String keyspaceName;
private final String migrationsPath;
public CassandraDataSourceMigration(String ip, String clusterName, Integer port, String keyspaceName, String migrationsPath) {
this.ip = ip;
this.clusterName = clusterName;
this.port = port;
this.keyspaceName = keyspaceName;
this.migrationsPath = migrationsPath;
}
// getters/setters
#Override
public void afterPropertiesSet() throws Exception {
final KeyspaceConfiguration keyspaceConfig = new KeyspaceConfiguration();
keyspaceConfig.setName(keyspaceName);
keyspaceConfig.getClusterConfig().setContactpoints(new String[]{ip});
if (port != null) {
keyspaceConfig.getClusterConfig().setPort(port);
}
final CassandraMigration migrationProcessor = new CassandraMigration();
migrationProcessor.setLocations(new String[]{migrationsPath});
migrationProcessor.setKeyspaceConfig(keyspaceConfig);
migrationProcessor.migrate();
}
}
application.properties
cassandra.ip=127.0.0.1
cassandra.cluster=My cluster
cassandra.keyspace=saya
cassandra.migration=classpath:db/migration
cassandra.port=9042
And the migration script is under resources/db/migration V1_0__Init_table.cql
We just upgraded to DB2 10.5 from 9.5, this process was working fine until the upgrade was performed on the server. When i run the jar file from a linux server, I get the following error however when i run the exact same code from eclipse on my windows computer, it works just fine! I am also getting a similar error if I calll this sp from DB2 control center. I am looking to know what is causing this and how can i fix this error?
SQL4306N Java stored procedure or user-defined function "ESADBM.GETNEXTID",
specific name "WHDBRMM_UTILS" could not call Java method "GetNextID",
signature "(Ljava/lang/String;[I)V". SQLSTATE=42724
Explanation:
The Java method given by the EXTERNAL NAME clause of a CREATE PROCEDURE
or CREATE FUNCTION statement could not be found. Its declared argument
list may not match what the database expects, or it may not be a
"public" instance method.
User response:
Ensure that a Java instance method exists with the "public" flag and the
expected argument list for this call.
sqlcode: -4306
sqlstate: 42724.
Here is the code:
package pkgUtil_v4_0_0_0;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import org.hibernate.exception.JDBCConnectionException;
public class DBSequence {
public static final String SEQ_CONTACTID = "ContactIDSeq";
public static final String SEQ_PROJECTID = "ProjectIDSeq";
public static final String SEQ_LOCATIONID = "LocationIDSeq";
public static final String SEQ_SOURCEID = "SourceIDSeq";
public static final String SEQ_SURVEYID = "SurveyIDSeq";
public static final String SEQ_LOGICALSURVEYID = "WageAreaIDSeq";
public static final String SEQ_WAGEDETAILID = "WageDetailIDSeq";
public static final String SEQ_ORGID = "OrgIDSeq";
public static final String SEQ_OFFICEID = "RegionNumberSeq";
public static final String SEQ_LETTERID = "LetterIDSeq";
public static final String SEQ_DODGEID = "DodgeIDSeq";
public static final String SEQ_CRAFTID = "CraftIDSeq";
public static final String SEQ_CRAFTTITLEID = "CraftTitleIDSeq";
public static final String SEQ_ANALYSTID = "AnalystIDSeq";
public static final String SEQ_LETTERTEMPLATEID = "LetterTemplateIDSeq";
public static final String SEQ_RECRATESID = "RecRatesIDSeq";
public static final String SEQ_BRIDGESCDID = "BridgeSCDIDSeq";
public static String drvr = "";
public static Connection con = null;
// utility function
public static int getNextId(Connection lcon, String sequence) throws SQLException {
Boolean bFlag;
PreparedStatement stmt = null;
int id = 0;
String sql = "select next value for esadbm." +
sequence + " from SYSIBM.sysdummy1";
// System.out.println("String = "+sequence);
stmt = lcon.prepareStatement(sql);
ResultSet resultSet = stmt.executeQuery();
if (resultSet.next()) {
id = resultSet.getInt(1);
}
resultSet.close();
stmt.close();
return id;
}
// Stored Procedure Entry Point
public static void getNextId(String sequence, int[] seq) throws SQLException, Exception {
System.out.println("String = "+sequence);
System.out.println("Array = "+seq);
if (drvr.length() == 0) {
drvr = "jdbc:default:connection";
con = DriverManager.getConnection(drvr);
}
drvr = "";
seq[0] = getNextId(con, sequence);
con.close();
}
// test procedure
public static void main(String args[])throws SQLException, Exception {
try {
System.out.println("Connecting to DB " + args[0]);
Class.forName("com.ibm.db2.jcc.DB2Driver");
drvr = "jdbc:db2:" + args[0];
// System.out.println(drvr+args[1] + args[2]);
con = DriverManager.getConnection("jdbc:db2:" + args[0], args[1],args[2]);
// System.out.println(con);
System.out.println("DB Connection Successful");
con = DriverManager.getConnection(drvr, args[1], args[2]);
Statement st = con.createStatement();
String query = "set schema = 'ESADBM'";
st.execute(query);
System.out.println("Getting ID");
int id = getNextId(con, SEQ_SOURCEID);
System.out.println("Returned : " + Integer.toString(id));
}
catch (ClassNotFoundException cnfe) {
cnfe.printStackTrace();
}
catch (SQLException sqle) {
sqle.printStackTrace();
}
catch (JDBCConnectionException e) {
System.out.println("Unable to connect to database");
e.printStackTrace();
}
}
}
Here is the stored procedure:
CREATE PROCEDURE "ESADBM "."GETNEXTID"
(
IN SEQUENCE CHARACTER(40),
OUT ID INTEGER
)
DYNAMIC RESULT SETS 0
SPECIFIC WHDBRA_UTILS
EXTERNAL NAME 'pkgUtil_v4_0_0_0.DBSequence!getNextId()'
LANGUAGE JAVA
PARAMETER STYLE JAVA
NOT DETERMINISTIC
FENCED THREADSAFE
MODIFIES SQL DATA
NO DBINFO;
Libraries for external routines, including Java classes and JAR files for Java routines, must be present in a certain location in the DB2 instance directory. When you upgrade your DB2 version, a new instance is created, but those libraries are not copied automatically (which, by the way, makes sense as there is a good chance that they need to be rebuilt).
The error message indicates that the instance cannot find the Java class file that implements GETNEXTID -- that would be DBSequence.class. The class needs to be copied to the sqllib/function directory in the DB2 10.5 instance home on the database server, as explained in the manual. You will probably need also to create pkgUtil_v4_0_0_0 under sqllib/function for the correct package structure. Make sure you compile the Java source using the same JDK version as the one used by the DB2 instance to run the program.
Once you do that, execute CALL SQLJ.REFRESH_CLASSES() in the DB2 client of your choice to make sure DB2 reloads the most current version. After that your stored procedure should work correctly.
Having said that, I don't really understand why you use such a convoluted way of retrieving a SQL sequence value.
Strange thing happened in Java Kingdom...
Long story short: I use Java API V3 to connect to QuickBooks and fetch the data form there (services for example).
Everything goes fine except the case when a service contains russian symbols (or probably non-latin symbols).
Here is Java code that does it (I know it's far from perfect)
package com.mde.test;
import static com.intuit.ipp.query.GenerateQuery.$;
import static com.intuit.ipp.query.GenerateQuery.select;
import java.util.LinkedList;
import java.util.List;
import com.intuit.ipp.core.Context;
import com.intuit.ipp.core.ServiceType;
import com.intuit.ipp.data.Item;
import com.intuit.ipp.exception.FMSException;
import com.intuit.ipp.query.GenerateQuery;
import com.intuit.ipp.security.OAuthAuthorizer;
import com.intuit.ipp.services.DataService;
import com.intuit.ipp.util.Config;
public class TestEncoding {
public static final String QBO_BASE_URL_SANDBOX = "https://sandbox-quickbooks.api.intuit.com/v3/company";
private static String consumerKey = "consumerkeycode";
private static String consumerSecret = "consumersecretcode";
private static String accessToken = "accesstokencode";
private static String accessTokenSecret = "accesstokensecretcode";
private static String appToken = "apptokencode";
private static String companyId = "companyidcode";
private static OAuthAuthorizer oauth = new OAuthAuthorizer(consumerKey, consumerSecret, accessToken, accessTokenSecret);
private static final int PAGING_STEP = 500;
public static void main(String[] args) throws FMSException {
List<Item> res = findAllServices(getDataService());
System.out.println(res.get(1).getName());
}
public static List<Item> findAllServices(DataService service) throws FMSException {
Item item = GenerateQuery.createQueryEntity(Item.class);
List<Item> res = new LinkedList<>();
for (int skip = 0; ; skip += PAGING_STEP) {
String query = select($(item)).skip(skip).take(PAGING_STEP).generate();
List<Item> items = (List<Item>)service.executeQuery(query).getEntities();
if (items.size() > 0)
res.addAll(items);
else
break;
}
System.out.println("All services fetched");
return res;
}
public static DataService getDataService() throws FMSException {
Context context = getContext();
if (context == null) {
System.out.println("Context is null, something wrong, dataService also will null.");
return null;
}
return getDataService(context);
}
private static Context getContext() {
try {
return new Context(oauth, appToken, ServiceType.QBO, companyId);
} catch (FMSException e) {
System.out.println("Context is not loaded");
return null;
}
}
protected static DataService getDataService(Context context) throws FMSException {
DataService service = new DataService(context);
Config.setProperty(Config.BASE_URL_QBO, QBO_BASE_URL_SANDBOX);
return new DataService(context);
}
}
This file is saved in UTF-8. And it prints something like
All services fetched
Сэрвыс, отнюдь
But! When I save this file in UTF-8 with BOM.... I get the correct data!
All services fetched
Сэрвыс, отнюдь
Does anybody can explain what is happening? :)
// I use Eclipse to run the code
You are fetching data from a system that doesn't share the same byte ordering as you, so when you save the file with BOM, it adds enough information in the file that future programs will read it in the remote system's byte ordering.
When you save it without BOM, it wrote the file in the remote system's byte ordering without any indication of the stored byte order, so when you read it you read it with the local system's (different) byte order. This jumbles up the bytes within the multi-byte characters, making the output appear as nonsense.
I am trying to fetch the xml file from flickr. Here is my code:
public static final String TAG = "FlickrFetchr";
public static final String ENDPOINT = "http://api.flickr.com/services/rest/";
public static final String API_KEY = "d4db9623ea909f4d2a01c8c9667fd378"; //secret=0c764276c114d52f
public static final String METHOD_GET_RECENT = "flickr.photos.getRecent";
public static final String PARAM_EXTRAS = "extras";
public static final String EXTRA_SMALL_URL = "url_s";
private static final String XML_PHOTO = "photo";
String url = Uri.parse(ENDPOINT).buildUpon().appendQueryParameter("method", METHOD_GET_RECENT)
.appendQueryParameter("api_key", API_KEY)
.appendQueryParameter(PARAM_EXTRAS, EXTRA_SMALL_URL)
.build().toString();
String xmlString = getUrl(url);
When i debug, i see that the url is (Docs)and it works, the xml file is there. But i am getting an exception that says "Unable to resolve host "api.flickr.com": No address associated with hostname".
Can anyone see the problem?
Thanks.
Seeing as it was the case and you requested it, here's it as an answer.
If you are running an emulator, please restart it. This happens a lot. That exception is also thrown when the device has no internet connection.