Activating virtualenv via Java ProcessBuilder - java

Getting the following when trying to programmatically activate Python's virtualenv via the code below:
java.io.IOException: Cannot run program "." (in directory "/Users/simeon.../..../reporting"): error=13, Permission denied
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at VirtualEnvCreateCmdTest.runCommandInDirectory(VirtualEnvCreateCmdTest.java:30)
at VirtualEnvCreateCmdTest.createVirtEnv(VirtualEnvCreateCmdTest.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at ......
Caused by: java.io.IOException: error=13, Permission denied
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 25 more
Code:
public class VirtualEnvCreateCmdTest {
private final static Logger LOG = LoggerFactory.getLogger(VirtualEnvCreateCmdTest.class);
private void runCommandInDirectory(String path,String ... command) throws Throwable
{
LOG.info("Running command '"+String.join(" ",command)+"' in path '"+path+"'");
ProcessBuilder builder = new ProcessBuilder(command)
.directory(new File(path))
.inheritIO();
Process pr = builder.start();
final String failureMsg = format("Failed to run '%s' in path '%s'. Got exit code: %d", join(" ",command), path, pr.exitValue());
LOG.info("prepared message {}: ", failureMsg);
if(!pr.waitFor(120, TimeUnit.SECONDS))
{
throw new Exception(failureMsg);
}
int output = IOUtils.copy(pr.getInputStream(), System.out);
int exitCode=pr.exitValue();
if(exitCode!=0)
throw new Exception(failureMsg);
}
#Test
public void createVirtEnv() throws Throwable {
String path = "/Users/simeon/.../reporting";
String [] commands = new String[]{".", "activate"};
//String [] commands = new String [] {"/bin/bash", "-c", ". /Users/simeon/..../venv2.7/bin/activate"};
runCommandInDirectory(path, commands);
}
Changing permissioning on the file, doesn't seem to work:
chmod u+x ./bin/activate
Doing it via /bin/bash still doesn't work though with a different error.
At the same time the following works fine on the command line:
. /Users/simeon/.../venv2.7/bin/activate
Any samples of how one would invoke python virtualenv activate command from within Java?
=== What worked: =====
The following ended up working for me:
private void runDjangoMigrate() throws Throwable {
final String REPORTING_PROJECT_LOCATION = "/Users/simeon.../.../reporting";
final String UNIX_SHELL_LOCATION = "/bin/bash";
final String PYTHON_VIRTUALENV_ACTIVATOR_COMMAND =". /Users/simeon.../.../venv2.7/bin/activate;";
final String PYTHON_VIRTUALENV_ACTIVATOR_COMMAND =". " + PYTHON_VIRTUALENV_ACTIVATE_SCRIPT_LOCATION + ";";
final String DJANGO_MANAGE_MODULE = " manage.py";
final String[] DJANGO_MIGRATE_COMMAND = new String[] { UNIX_SHELL_LOCATION, "-c", PYTHON_VIRTUALENV_ACTIVATOR_COMMAND
+ PYTHON_INTERPRETER + DJANGO_MANAGE_MODULE + " migrate --noinput --fake-initial" };
runCommandInDirectory(REPORTING_PROJECT_LOCATION, DJANGO_MIGRATE_COMMAND);
}
private void runCommandInDirectory(String path, String... command) throws Throwable {
LOG.info(format("Running '%s' command in path '%s'", join(" ",command),path));
ProcessBuilder builder = new ProcessBuilder(command).directory(new File(path)).inheritIO();
Process pr = null;
pr = builder.start();
IOUtils.copy(pr.getInputStream(), System.out);
boolean terminated = pr.waitFor(SPAWNED_PYTHON_PROCESS_TTL_SEC, SECONDS);
int exitCode = -1;
if (terminated) {
exitCode = pr.exitValue();
}
if (exitCode != 0) {
final String failureMsg = format("Failed to run '%s' in path '%s'. Got exit code: %d", join(" ", command),
path, pr.exitValue());
throw new Exception(failureMsg);
}
}
and producing the following expected output:
[java] System check identified some issues:
[java]
[java] WARNINGS:
[java] ?: (urls.W005) URL namespace 'admin' isn't unique. You may not be able to reverse all URLs in this namespace
[java] Building permissions...
[java] Operations to perform:
[java] Apply all migrations: admin, auth, contenttypes, sessions
[java] Running migrations:
[java] No migrations to apply.
[java] System check identified some issues:
[java]
[java] WARNINGS:
[java] ?: (urls.W005) URL namespace 'admin' isn't unique. You may not be able to reverse all URLs in this namespace
[java] Building permissions...
[java] User exists, exiting normally due to --preserve
[java] System check identified some issues:
[java]

You can activate a Python virtual environment in bash (and perhaps zsh) and Python, but not in Java or any other environment. The main thing to understand is that virtual environment activation doesn't do some system magic — the activation script just changes the current environment, and Python virtual environments are prepared to changes shell or Python but nothing else.
In terms of Java it means that when you call runCommandInDirectory() it runs a new shell, that new shell briefly activates a virtual env, but then the shell exits and all changes that "activates" the virtual env are gone.
This in turn means that if you need to run some shell or Python commands in a virtual env you have to activate the environment in every runCommandInDirectory() invocation:
String [] commands1 = new String [] {"/bin/bash", "-c", ". /Users/simeon/..../venv2.7/bin/activate; script1.sh"};
runCommandInDirectory(path, commands)
String [] commands2 = new String [] {"/bin/bash", "-c", ". /Users/simeon/..../venv2.7/bin/activate; script2.sh"};
runCommandInDirectory(path, commands)
For Python scripts it's a bit simpler as you can run python from the environment and it automagically activates the environment:
String [] commands = new String [] {"/Users/simeon/..../venv2.7/bin/python", "script.py"};
runCommandInDirectory(path, commands)

Related

Ensure Install4j Uses only its bundled jre and never Java found by path

question
How can I ensure my install4j installer always finds only its java?
Can I create a top level installer which installs JRE to tmp, sets env variables and then starts the actual installer?
Can I load a vm file during installation?
problem
Install4j finds java 1.7 during install which impacts custom code preventing successful installation. I see found java7 prior to file deployment - ok expected given the JRE hasn't yet been unpacked.
evidence
I created a simple installer and see the following:
BEFORE
PATH=/opt/tools/Java/jdk1.7.0_79/bin:...
JAVA_HOME=/opt/tools/Java/jdk1.7.0_79
...
ENV [JAVA_HOME] /opt/tools/Java/jdk1.7.0_79
ENV [PATH] /opt/tools/Java/jdk1.7.0_79/bin:...
installer details
envTest.install4j
Optional customer install script reporting found java prior at execution start
echo BEFORE
echo PATH=$PATH
echo JAVA_HOME=$JAVA_HOME
echo Version: java -version
Run script reporting env after installer deployed jre
`
import java.util.Map;
import java.util.Set;
import java.util.SortedMap;
import java.util.TreeMap;
Map<String, String> envMap = System.getenv();
SortedMap<String, String> sortedEnvMap = new TreeMap<String, String>(envMap);
Set<String> keySet = sortedEnvMap.keySet();
for (String key : keySet) {
String value = envMap.get(key);
Util.logInfo(this,"ENV [" + key + "] " + value);
}
return true;
Actually, this turned our to be a problem with my custom code. The custom code launches an install4j generated executable via java. When launched on command line with wrong java found first, the launcher uses only its own java. When launched from my extension it fails.
Solution - set java in my extension:
private File getInstalledJREDir() {
return new File(installationDir, "jre");
}
private String addJREToFrontOfPathVar() {
File jreBinDir = new File(getInstalledJREDir(), "bin");
String path = System.getenv().get("PATH");
if (null == path) {
path = jreBinDir.getAbsolutePath();
} else {
path = jreBinDir.getAbsolutePath() + File.pathSeparator + path;
}
return path;
}
/**
* Start Laucnher and block until it starts or timeout reached
* #throws AutoRunException
*/
public void run() throws AutoRunException, IOException, InterruptedException {
notifier.setPhase("Starting Agent");
// Set Directories
File dataDir = new File(installationDir.getParentFile(), "data-agent");
File agentLog = new File(logDir,"agent.log");
if (! isWindows()) {
File agent = new File(installationDir, "bin/launcherExecutable");
CmdExecutor ce = new CmdExecutor(agent, agentLog);
// Ensure our installed JRE found 1st - PLAT-38833
ce.updateEnvironmentVariable("JAVA_HOME", getInstalledJREDir().getAbsolutePath());
ce.updateEnvironmentVariable("PATH", addJREToFrontOfPathVar());
ce.setWorkingDir(installationDir);
ce.setArgLine(String.format("--datadir %s", dataDir.getAbsolutePath()));
notifier.logInfo("Starting " + agent + " with " + ce.getArgLine());
if (! ce.run(true) ) {
throw new AutoRunException("Agent failed to start " + ce.getOutput());
}

How to pass a string value of length greater than 255 to a processbuilder in java

I am having a command having a length greater than 255. The command is a Sqoop (Hadoop) import which loads data into the hive from a RDBMS. I am using the ProcessBuilder API which provide two constructors
ProcessBuilder(String[] command)
ProcessBuilder(List<String> command)
I tried passing the command in both the ways as mentioned above
ssh hddev-chost-01-01 "sqoop import -D oraoop.disabled=true -D mapred.child.java.opts='\-Djava.security.egd=file:/dev/../dev/urandom' --connect 'jdbc:oracle:thin:#(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=maildude.com)(PORT=1533))(CONNECT_DATA=(SERVICE_NAME=SDA.COM)(Server=Dedicated)))' --username user --password pass --query \"select INVENTORY_ITEM_ID,ORGANIZATION_ID,LAST_UPDATE_DATE,LAST_UPDATED_BY,CREATION_DATE,CREATED_BY,LAST_UPDATE_LOGIN,SUMMARY_FLAG,ENABLED_FLAG,START_DATE_ACTIVE,END_DATE_ACTIVE,DESCRIPTION,BUYER_ID,ACCOUNTING_RULE_ID,INVOICING_RULE_ID,SEGMENT1,SEGMENT2,SEGMENT3,SEGMENT4,SEGMENT5,SEGMENT6,SEGMENT7,SEGMENT8,SEGMENT9,SEGMENT10,SEGMENT11,SEGMENT12,SEGMENT13,SEGMENT14,SEGMENT15,SEGMENT16,SEGMENT17,SEGMENT18,SEGMENT19,SEGMENT20,ATTRIBUTE_CATEGORY,ATTRIBUTE1,ATTRIBUTE2,ATTRIBUTE3,ATTRIBUTE4,ATTRIBUTE5,ATTRIBUTE6,ATTRIBUTE7,ATTRIBUTE8,ATTRIBUTE9,ATTRIBUTE10,ATTRIBUTE11,ATTRIBUTE12,ATTRIBUTE13,ATTRIBUTE14,ATTRIBUTE15,PURCHASING_ITEM_FLAG,SHIPPABLE_ITEM_FLAG,CUSTOMER_ORDER_FLAG,INTERNAL_ORDER_FLAG,SERVICE_ITEM_FLAG,INVENTORY_ITEM_FLAG,ENG_ITEM_FLAG,INVENTORY_ASSET_FLAG,PURCHASING_ENABLED_FLAG,CUSTOMER_ORDER_ENABLED_FLAG,INTERNAL_ORDER_ENABLED_FLAG,SO_TRANSACTIONS_FLAG,MTL_TRANSACTIONS_ENABLED_FLAG,STOCK_ENABLED_FLAG,BOM_ENABLED_FLAG,BUILD_IN_WIP_FLAG,REVISION_QTY_CONTROL_CODE,ITEM_CATALOG_GROUP_ID,CATALOG_STATUS_FLAG,RETURNABLE_FLAG,DEFAULT_SHIPPING_ORG,COLLATERAL_FLAG,TAXABLE_FLAG,QTY_RCV_EXCEPTION_CODE,ALLOW_ITEM_DESC_UPDATE_FLAG,INSPECTION_REQUIRED_FLAG,RECEIPT_REQUIRED_FLAG,MARKET_PRICE,HAZARD_CLASS_ID,RFQ_REQUIRED_FLAG,QTY_RCV_TOLERANCE,LIST_PRICE_PER_UNIT,UN_NUMBER_ID,PRICE_TOLERANCE_PERCENT,ASSET_CATEGORY_ID,ROUNDING_FACTOR,UNIT_OF_ISSUE,ENFORCE_SHIP_TO_LOCATION_CODE,ALLOW_SUBSTITUTE_RECEIPTS_FLAG,ALLOW_UNORDERED_RECEIPTS_FLAG,ALLOW_EXPRESS_DELIVERY_FLAG,DAYS_EARLY_RECEIPT_ALLOWED,DAYS_LATE_RECEIPT_ALLOWED,RECEIPT_DAYS_EXCEPTION_CODE,RECEIVING_ROUTING_ID,INVOICE_CLOSE_TOLERANCE,RECEIVE_CLOSE_TOLERANCE,AUTO_LOT_ALPHA_PREFIX,START_AUTO_LOT_NUMBER,LOT_CONTROL_CODE,SHELF_LIFE_CODE,SHELF_LIFE_DAYS,SERIAL_NUMBER_CONTROL_CODE,START_AUTO_SERIAL_NUMBER,AUTO_SERIAL_ALPHA_PREFIX,SOURCE_TYPE,SOURCE_ORGANIZATION_ID,SOURCE_SUBINVENTORY,EXPENSE_ACCOUNT,ENCUMBRANCE_ACCOUNT,RESTRICT_SUBINVENTORIES_CODE,UNIT_WEIGHT,WEIGHT_UOM_CODE,VOLUME_UOM_CODE,UNIT_VOLUME,RESTRICT_LOCATORS_CODE,LOCATION_CONTROL_CODE,SHRINKAGE_RATE,ACCEPTABLE_EARLY_DAYS,PLANNING_TIME_FENCE_CODE,DEMAND_TIME_FENCE_CODE,LEAD_TIME_LOT_SIZE,STD_LOT_SIZE,CUM_MANUFACTURING_LEAD_TIME,OVERRUN_PERCENTAGE,MRP_CALCULATE_ATP_FLAG,ACCEPTABLE_RATE_INCREASE,ACCEPTABLE_RATE_DECREASE,CUMULATIVE_TOTAL_LEAD_TIME,PLANNING_TIME_FENCE_DAYS,DEMAND_TIME_FENCE_DAYS,END_ASSEMBLY_PEGGING_FLAG,REPETITIVE_PLANNING_FLAG,PLANNING_EXCEPTION_SET,BOM_ITEM_TYPE,PICK_COMPONENTS_FLAG,REPLENISH_TO_ORDER_FLAG,BASE_ITEM_ID,ATP_COMPONENTS_FLAG,ATP_FLAG,FIXED_LEAD_TIME,VARIABLE_LEAD_TIME,WIP_SUPPLY_LOCATOR_ID,WIP_SUPPLY_TYPE,WIP_SUPPLY_SUBINVENTORY,PRIMARY_UOM_CODE,PRIMARY_UNIT_OF_MEASURE,ALLOWED_UNITS_LOOKUP_CODE,COST_OF_SALES_ACCOUNT,SALES_ACCOUNT,DEFAULT_INCLUDE_IN_ROLLUP_FLAG,INVENTORY_ITEM_STATUS_CODE,INVENTORY_PLANNING_CODE,PLANNER_CODE,PLANNING_MAKE_BUY_CODE,FIXED_LOT_MULTIPLIER,ROUNDING_CONTROL_TYPE,CARRYING_COST,POSTPROCESSING_LEAD_TIME,PREPROCESSING_LEAD_TIME,FULL_LEAD_TIME,ORDER_COST,MRP_SAFETY_STOCK_PERCENT,MRP_SAFETY_STOCK_CODE,MIN_MINMAX_QUANTITY,MAX_MINMAX_QUANTITY,MINIMUM_ORDER_QUANTITY,FIXED_ORDER_QUANTITY,FIXED_DAYS_SUPPLY,MAXIMUM_ORDER_QUANTITY,ATP_RULE_ID,PICKING_RULE_ID,RESERVABLE_TYPE,POSITIVE_MEASUREMENT_ERROR,NEGATIVE_MEASUREMENT_ERROR,ENGINEERING_ECN_CODE,ENGINEERING_ITEM_ID,ENGINEERING_DATE,SERVICE_STARTING_DELAY,VENDOR_WARRANTY_FLAG,SERVICEABLE_COMPONENT_FLAG,SERVICEABLE_PRODUCT_FLAG,BASE_WARRANTY_SERVICE_ID,PAYMENT_TERMS_ID,PREVENTIVE_MAINTENANCE_FLAG,PRIMARY_SPECIALIST_ID,SECONDARY_SPECIALIST_ID,SERVICEABLE_ITEM_CLASS_ID,TIME_BILLABLE_FLAG,MATERIAL_BILLABLE_FLAG,EXPENSE_BILLABLE_FLAG,PRORATE_SERVICE_FLAG,COVERAGE_SCHEDULE_ID,SERVICE_DURATION_PERIOD_CODE,SERVICE_DURATION,WARRANTY_VENDOR_ID,MAX_WARRANTY_AMOUNT,RESPONSE_TIME_PERIOD_CODE,RESPONSE_TIME_VALUE,NEW_REVISION_CODE,INVOICEABLE_ITEM_FLAG,TAX_CODE,INVOICE_ENABLED_FLAG,MUST_USE_APPROVED_VENDOR_FLAG,REQUEST_ID,PROGRAM_APPLICATION_ID,PROGRAM_ID,PROGRAM_UPDATE_DATE,OUTSIDE_OPERATION_FLAG,OUTSIDE_OPERATION_UOM_TYPE,SAFETY_STOCK_BUCKET_DAYS,AUTO_REDUCE_MPS,COSTING_ENABLED_FLAG,AUTO_CREATED_CONFIG_FLAG,CYCLE_COUNT_ENABLED_FLAG,ITEM_TYPE,MODEL_CONFIG_CLAUSE_NAME,SHIP_MODEL_COMPLETE_FLAG,MRP_PLANNING_CODE,RETURN_INSPECTION_REQUIREMENT,ATO_FORECAST_CONTROL,RELEASE_TIME_FENCE_CODE,RELEASE_TIME_FENCE_DAYS,CONTAINER_ITEM_FLAG,VEHICLE_ITEM_FLAG,MAXIMUM_LOAD_WEIGHT,MINIMUM_FILL_PERCENT,CONTAINER_TYPE_CODE,INTERNAL_VOLUME,WH_UPDATE_DATE,PRODUCT_FAMILY_ITEM_ID,GLOBAL_ATTRIBUTE_CATEGORY,GLOBAL_ATTRIBUTE1,GLOBAL_ATTRIBUTE2,GLOBAL_ATTRIBUTE3,GLOBAL_ATTRIBUTE4,GLOBAL_ATTRIBUTE5,GLOBAL_ATTRIBUTE6,GLOBAL_ATTRIBUTE7,GLOBAL_ATTRIBUTE8,GLOBAL_ATTRIBUTE9,GLOBAL_ATTRIBUTE10,PURCHASING_TAX_CODE,OVERCOMPLETION_TOLERANCE_TYPE,OVERCOMPLETION_TOLERANCE_VALUE,EFFECTIVITY_CONTROL,CHECK_SHORTAGES_FLAG,OVER_SHIPMENT_TOLERANCE,UNDER_SHIPMENT_TOLERANCE,OVER_RETURN_TOLERANCE,UNDER_RETURN_TOLERANCE,EQUIPMENT_TYPE,RECOVERED_PART_DISP_CODE,DEFECT_TRACKING_ON_FLAG,USAGE_ITEM_FLAG,EVENT_FLAG,ELECTRONIC_FLAG,DOWNLOADABLE_FLAG,VOL_DISCOUNT_EXEMPT_FLAG,COUPON_EXEMPT_FLAG,COMMS_NL_TRACKABLE_FLAG,ASSET_CREATION_CODE,COMMS_ACTIVATION_REQD_FLAG,ORDERABLE_ON_WEB_FLAG,BACK_ORDERABLE_FLAG,WEB_STATUS,INDIVISIBLE_FLAG,DIMENSION_UOM_CODE,UNIT_LENGTH,UNIT_WIDTH,UNIT_HEIGHT,BULK_PICKED_FLAG,LOT_STATUS_ENABLED,DEFAULT_LOT_STATUS_ID,SERIAL_STATUS_ENABLED,DEFAULT_SERIAL_STATUS_ID,LOT_SPLIT_ENABLED,LOT_MERGE_ENABLED,INVENTORY_CARRY_PENALTY,OPERATION_SLACK_PENALTY,FINANCING_ALLOWED_FLAG,EAM_ITEM_TYPE,EAM_ACTIVITY_TYPE_CODE,EAM_ACTIVITY_CAUSE_CODE,EAM_ACT_NOTIFICATION_FLAG,EAM_ACT_SHUTDOWN_STATUS,DUAL_UOM_CONTROL,SECONDARY_UOM_CODE,DUAL_UOM_DEVIATION_HIGH,DUAL_UOM_DEVIATION_LOW,CONTRACT_ITEM_TYPE_CODE,SUBSCRIPTION_DEPEND_FLAG,SERV_REQ_ENABLED_CODE,SERV_BILLING_ENABLED_FLAG,SERV_IMPORTANCE_LEVEL,PLANNED_INV_POINT_FLAG,LOT_TRANSLATE_ENABLED,DEFAULT_SO_SOURCE_TYPE,CREATE_SUPPLY_FLAG,SUBSTITUTION_WINDOW_CODE,SUBSTITUTION_WINDOW_DAYS,IB_ITEM_INSTANCE_CLASS,CONFIG_MODEL_TYPE,LOT_SUBSTITUTION_ENABLED,MINIMUM_LICENSE_QUANTITY,EAM_ACTIVITY_SOURCE_CODE,LIFECYCLE_ID,CURRENT_PHASE_ID,OBJECT_VERSION_NUMBER,CDB_DATA_SOURCE,CDB_ENQUEUE_TIME,CDB_DEQUEUE_TIME,CDB_ENQUEUE_SEQ from MTL_SYSTEM_ITEMS_B where \\\$CONDITIONS \" --target-dir /app/dev/SmartAnalytics/sqoop_temp//MTL_SYSTEM_ITEMS_B -m 4 --split-by INVENTORY_ITEM_ID --hive-import --hive-overwrite --hive-table installbase.MTL_SYSTEM_ITEMS_B --null-string '\\\\N' --null-non-string '\\\\N' --hive-delims-replacement '<EOL>' --map-column-hive INVENTORY_ITEM_ID=BIGINT,ORGANIZATION_ID=BIGINT,LAST_UPDATE_DATE=TIMESTAMP,LAST_UPDATED_BY=BIGINT,CREATION_DATE=TIMESTAMP,CREATED_BY=BIGINT,LAST_UPDATE_LOGIN=BIGINT,SUMMARY_FLAG=STRING,ENABLED_FLAG=STRING,START_DATE_ACTIVE=TIMESTAMP,END_DATE_ACTIVE=TIMESTAMP,DESCRIPTION=STRING,BUYER_ID=BIGINT,ACCOUNTING_RULE_ID=BIGINT,INVOICING_RULE_ID=BIGINT,SEGMENT1=STRING,SEGMENT2=STRING,SEGMENT3=STRING,SEGMENT4=STRING,SEGMENT5=STRING,SEGMENT6=STRING,SEGMENT7=STRING,SEGMENT8=STRING,SEGMENT9=STRING,SEGMENT10=STRING,SEGMENT11=STRING,SEGMENT12=STRING,SEGMENT13=STRING,SEGMENT14=STRING,SEGMENT15=STRING,SEGMENT16=STRING,SEGMENT17=STRING,SEGMENT18=STRING,SEGMENT19=STRING,SEGMENT20=STRING,ATTRIBUTE_CATEGORY=STRING,ATTRIBUTE1=STRING,ATTRIBUTE2=STRING,ATTRIBUTE3=STRING,ATTRIBUTE4=STRING,ATTRIBUTE5=STRING,ATTRIBUTE6=STRING,ATTRIBUTE7=STRING,ATTRIBUTE8=STRING,ATTRIBUTE9=STRING,ATTRIBUTE10=STRING,ATTRIBUTE11=STRING,ATTRIBUTE12=STRING,ATTRIBUTE13=STRING,ATTRIBUTE14=STRING,ATTRIBUTE15=STRING,PURCHASING_ITEM_FLAG=STRING,SHIPPABLE_ITEM_FLAG=STRING,CUSTOMER_ORDER_FLAG=STRING,INTERNAL_ORDER_FLAG=STRING,SERVICE_ITEM_FLAG=STRING,INVENTORY_ITEM_FLAG=STRING,ENG_ITEM_FLAG=STRING,INVENTORY_ASSET_FLAG=STRING,PURCHASING_ENABLED_FLAG=STRING,CUSTOMER_ORDER_ENABLED_FLAG=STRING,INTERNAL_ORDER_ENABLED_FLAG=STRING,SO_TRANSACTIONS_FLAG=STRING,MTL_TRANSACTIONS_ENABLED_FLAG=STRING,STOCK_ENABLED_FLAG=STRING,BOM_ENABLED_FLAG=STRING,BUILD_IN_WIP_FLAG=STRING,REVISION_QTY_CONTROL_CODE=BIGINT,ITEM_CATALOG_GROUP_ID=BIGINT,CATALOG_STATUS_FLAG=STRING,RETURNABLE_FLAG=STRING,DEFAULT_SHIPPING_ORG=BIGINT,COLLATERAL_FLAG=STRING,TAXABLE_FLAG=STRING,QTY_RCV_EXCEPTION_CODE=STRING,ALLOW_ITEM_DESC_UPDATE_FLAG=STRING,INSPECTION_REQUIRED_FLAG=STRING,RECEIPT_REQUIRED_FLAG=STRING,MARKET_PRICE=BIGINT,HAZARD_CLASS_ID=BIGINT,RFQ_REQUIRED_FLAG=STRING,QTY_RCV_TOLERANCE=BIGINT,LIST_PRICE_PER_UNIT=BIGINT,UN_NUMBER_ID=BIGINT,PRICE_TOLERANCE_PERCENT=BIGINT,ASSET_CATEGORY_ID=BIGINT,ROUNDING_FACTOR=BIGINT,UNIT_OF_ISSUE=STRING,ENFORCE_SHIP_TO_LOCATION_CODE=STRING,ALLOW_SUBSTITUTE_RECEIPTS_FLAG=STRING,ALLOW_UNORDERED_RECEIPTS_FLAG=STRING,ALLOW_EXPRESS_DELIVERY_FLAG=STRING,DAYS_EARLY_RECEIPT_ALLOWED=BIGINT,DAYS_LATE_RECEIPT_ALLOWED=BIGINT,RECEIPT_DAYS_EXCEPTION_CODE=STRING,RECEIVING_ROUTING_ID=BIGINT,INVOICE_CLOSE_TOLERANCE=BIGINT,RECEIVE_CLOSE_TOLERANCE=BIGINT,AUTO_LOT_ALPHA_PREFIX=STRING,START_AUTO_LOT_NUMBER=STRING,LOT_CONTROL_CODE=BIGINT,SHELF_LIFE_CODE=BIGINT,SHELF_LIFE_DAYS=BIGINT,SERIAL_NUMBER_CONTROL_CODE=BIGINT,START_AUTO_SERIAL_NUMBER=STRING,AUTO_SERIAL_ALPHA_PREFIX=STRING,SOURCE_TYPE=BIGINT,SOURCE_ORGANIZATION_ID=BIGINT,SOURCE_SUBINVENTORY=STRING,EXPENSE_ACCOUNT=BIGINT,ENCUMBRANCE_ACCOUNT=BIGINT,RESTRICT_SUBINVENTORIES_CODE=BIGINT,UNIT_WEIGHT=BIGINT,WEIGHT_UOM_CODE=STRING,VOLUME_UOM_CODE=STRING,UNIT_VOLUME=BIGINT,RESTRICT_LOCATORS_CODE=BIGINT,LOCATION_CONTROL_CODE=BIGINT,SHRINKAGE_RATE=BIGINT,ACCEPTABLE_EARLY_DAYS=BIGINT,PLANNING_TIME_FENCE_CODE=BIGINT,DEMAND_TIME_FENCE_CODE=BIGINT,LEAD_TIME_LOT_SIZE=BIGINT,STD_LOT_SIZE=BIGINT,CUM_MANUFACTURING_LEAD_TIME=BIGINT,OVERRUN_PERCENTAGE=BIGINT,MRP_CALCULATE_ATP_FLAG=STRING,ACCEPTABLE_RATE_INCREASE=BIGINT,ACCEPTABLE_RATE_DECREASE=BIGINT,CUMULATIVE_TOTAL_LEAD_TIME=BIGINT,PLANNING_TIME_FENCE_DAYS=BIGINT,DEMAND_TIME_FENCE_DAYS=BIGINT,END_ASSEMBLY_PEGGING_FLAG=STRING,REPETITIVE_PLANNING_FLAG=STRING,PLANNING_EXCEPTION_SET=STRING,BOM_ITEM_TYPE=BIGINT,PICK_COMPONENTS_FLAG=STRING,REPLENISH_TO_ORDER_FLAG=STRING,BASE_ITEM_ID=BIGINT,ATP_COMPONENTS_FLAG=STRING,ATP_FLAG=STRING,FIXED_LEAD_TIME=BIGINT,VARIABLE_LEAD_TIME=BIGINT,WIP_SUPPLY_LOCATOR_ID=BIGINT,WIP_SUPPLY_TYPE=BIGINT,WIP_SUPPLY_SUBINVENTORY=STRING,PRIMARY_UOM_CODE=STRING,PRIMARY_UNIT_OF_MEASURE=STRING,ALLOWED_UNITS_LOOKUP_CODE=BIGINT,COST_OF_SALES_ACCOUNT=BIGINT,SALES_ACCOUNT=BIGINT,DEFAULT_INCLUDE_IN_ROLLUP_FLAG=STRING,INVENTORY_ITEM_STATUS_CODE=STRING,INVENTORY_PLANNING_CODE=BIGINT,PLANNER_CODE=STRING,PLANNING_MAKE_BUY_CODE=BIGINT,FIXED_LOT_MULTIPLIER=BIGINT,ROUNDING_CONTROL_TYPE=BIGINT,CARRYING_COST=BIGINT,POSTPROCESSING_LEAD_TIME=BIGINT,PREPROCESSING_LEAD_TIME=BIGINT,FULL_LEAD_TIME=BIGINT,ORDER_COST=BIGINT,MRP_SAFETY_STOCK_PERCENT=BIGINT,MRP_SAFETY_STOCK_CODE=BIGINT,MIN_MINMAX_QUANTITY=BIGINT,MAX_MINMAX_QUANTITY=BIGINT,MINIMUM_ORDER_QUANTITY=BIGINT,FIXED_ORDER_QUANTITY=BIGINT,FIXED_DAYS_SUPPLY=BIGINT,MAXIMUM_ORDER_QUANTITY=BIGINT,ATP_RULE_ID=BIGINT,PICKING_RULE_ID=BIGINT,RESERVABLE_TYPE=BIGINT,POSITIVE_MEASUREMENT_ERROR=BIGINT,NEGATIVE_MEASUREMENT_ERROR=BIGINT,ENGINEERING_ECN_CODE=STRING,ENGINEERING_ITEM_ID=BIGINT,ENGINEERING_DATE=TIMESTAMP,SERVICE_STARTING_DELAY=BIGINT,VENDOR_WARRANTY_FLAG=STRING,SERVICEABLE_COMPONENT_FLAG=STRING,SERVICEABLE_PRODUCT_FLAG=STRING,BASE_WARRANTY_SERVICE_ID=BIGINT,PAYMENT_TERMS_ID=BIGINT,PREVENTIVE_MAINTENANCE_FLAG=STRING,PRIMARY_SPECIALIST_ID=BIGINT,SECONDARY_SPECIALIST_ID=BIGINT,SERVICEABLE_ITEM_CLASS_ID=BIGINT,TIME_BILLABLE_FLAG=STRING,MATERIAL_BILLABLE_FLAG=STRING,EXPENSE_BILLABLE_FLAG=STRING,PRORATE_SERVICE_FLAG=STRING,COVERAGE_SCHEDULE_ID=BIGINT,SERVICE_DURATION_PERIOD_CODE=STRING,SERVICE_DURATION=BIGINT,WARRANTY_VENDOR_ID=BIGINT,MAX_WARRANTY_AMOUNT=BIGINT,RESPONSE_TIME_PERIOD_CODE=STRING,RESPONSE_TIME_VALUE=BIGINT,NEW_REVISION_CODE=STRING,INVOICEABLE_ITEM_FLAG=STRING,TAX_CODE=STRING,INVOICE_ENABLED_FLAG=STRING,MUST_USE_APPROVED_VENDOR_FLAG=STRING,REQUEST_ID=BIGINT,PROGRAM_APPLICATION_ID=BIGINT,PROGRAM_ID=BIGINT,PROGRAM_UPDATE_DATE=TIMESTAMP,OUTSIDE_OPERATION_FLAG=STRING,OUTSIDE_OPERATION_UOM_TYPE=STRING,SAFETY_STOCK_BUCKET_DAYS=BIGINT,AUTO_REDUCE_MPS=BIGINT,COSTING_ENABLED_FLAG=STRING,AUTO_CREATED_CONFIG_FLAG=STRING,CYCLE_COUNT_ENABLED_FLAG=STRING,ITEM_TYPE=STRING,MODEL_CONFIG_CLAUSE_NAME=STRING,SHIP_MODEL_COMPLETE_FLAG=STRING,MRP_PLANNING_CODE=BIGINT,RETURN_INSPECTION_REQUIREMENT=BIGINT,ATO_FORECAST_CONTROL=BIGINT,RELEASE_TIME_FENCE_CODE=BIGINT,RELEASE_TIME_FENCE_DAYS=BIGINT,CONTAINER_ITEM_FLAG=STRING,VEHICLE_ITEM_FLAG=STRING,MAXIMUM_LOAD_WEIGHT=BIGINT,MINIMUM_FILL_PERCENT=BIGINT,CONTAINER_TYPE_CODE=STRING,INTERNAL_VOLUME=BIGINT,WH_UPDATE_DATE=TIMESTAMP,PRODUCT_FAMILY_ITEM_ID=BIGINT,GLOBAL_ATTRIBUTE_CATEGORY=STRING,GLOBAL_ATTRIBUTE1=STRING,GLOBAL_ATTRIBUTE2=STRING,GLOBAL_ATTRIBUTE3=STRING,GLOBAL_ATTRIBUTE4=STRING,GLOBAL_ATTRIBUTE5=STRING,GLOBAL_ATTRIBUTE6=STRING,GLOBAL_ATTRIBUTE7=STRING,GLOBAL_ATTRIBUTE8=STRING,GLOBAL_ATTRIBUTE9=STRING,GLOBAL_ATTRIBUTE10=STRING,PURCHASING_TAX_CODE=STRING,OVERCOMPLETION_TOLERANCE_TYPE=BIGINT,OVERCOMPLETION_TOLERANCE_VALUE=BIGINT,EFFECTIVITY_CONTROL=BIGINT,CHECK_SHORTAGES_FLAG=STRING,OVER_SHIPMENT_TOLERANCE=BIGINT,UNDER_SHIPMENT_TOLERANCE=BIGINT,OVER_RETURN_TOLERANCE=BIGINT,UNDER_RETURN_TOLERANCE=BIGINT,EQUIPMENT_TYPE=BIGINT,RECOVERED_PART_DISP_CODE=STRING,DEFECT_TRACKING_ON_FLAG=STRING,USAGE_ITEM_FLAG=STRING,EVENT_FLAG=STRING,ELECTRONIC_FLAG=STRING,DOWNLOADABLE_FLAG=STRING,VOL_DISCOUNT_EXEMPT_FLAG=STRING,COUPON_EXEMPT_FLAG=STRING,COMMS_NL_TRACKABLE_FLAG=STRING,ASSET_CREATION_CODE=STRING,COMMS_ACTIVATION_REQD_FLAG=STRING,ORDERABLE_ON_WEB_FLAG=STRING,BACK_ORDERABLE_FLAG=STRING,WEB_STATUS=STRING,INDIVISIBLE_FLAG=STRING,DIMENSION_UOM_CODE=STRING,UNIT_LENGTH=BIGINT,UNIT_WIDTH=BIGINT,UNIT_HEIGHT=BIGINT,BULK_PICKED_FLAG=STRING,LOT_STATUS_ENABLED=STRING,DEFAULT_LOT_STATUS_ID=BIGINT,SERIAL_STATUS_ENABLED=STRING,DEFAULT_SERIAL_STATUS_ID=BIGINT,LOT_SPLIT_ENABLED=STRING,LOT_MERGE_ENABLED=STRING,INVENTORY_CARRY_PENALTY=BIGINT,OPERATION_SLACK_PENALTY=BIGINT,FINANCING_ALLOWED_FLAG=STRING,EAM_ITEM_TYPE=BIGINT,EAM_ACTIVITY_TYPE_CODE=STRING,EAM_ACTIVITY_CAUSE_CODE=STRING,EAM_ACT_NOTIFICATION_FLAG=STRING,EAM_ACT_SHUTDOWN_STATUS=STRING,DUAL_UOM_CONTROL=BIGINT,SECONDARY_UOM_CODE=STRING,DUAL_UOM_DEVIATION_HIGH=BIGINT,DUAL_UOM_DEVIATION_LOW=BIGINT,CONTRACT_ITEM_TYPE_CODE=STRING,SUBSCRIPTION_DEPEND_FLAG=STRING,SERV_REQ_ENABLED_CODE=STRING,SERV_BILLING_ENABLED_FLAG=STRING,SERV_IMPORTANCE_LEVEL=BIGINT,PLANNED_INV_POINT_FLAG=STRING,LOT_TRANSLATE_ENABLED=STRING,DEFAULT_SO_SOURCE_TYPE=STRING,CREATE_SUPPLY_FLAG=STRING,SUBSTITUTION_WINDOW_CODE=BIGINT,SUBSTITUTION_WINDOW_DAYS=BIGINT,IB_ITEM_INSTANCE_CLASS=STRING,CONFIG_MODEL_TYPE=STRING,LOT_SUBSTITUTION_ENABLED=STRING,MINIMUM_LICENSE_QUANTITY=BIGINT,EAM_ACTIVITY_SOURCE_CODE=STRING,LIFECYCLE_ID=BIGINT,CURRENT_PHASE_ID=BIGINT,OBJECT_VERSION_NUMBER=BIGINT,CDB_DATA_SOURCE=STRING,CDB_ENQUEUE_TIME=TIMESTAMP,CDB_DEQUEUE_TIME=TIMESTAMP,CDB_ENQUEUE_SEQ=BIGINT --boundary-query 'select (select min(INVENTORY_ITEM_ID) from MTL_SYSTEM_ITEMS_B) as minid ,(select max(INVENTORY_ITEM_ID) from MTL_SYSTEM_ITEMS_B) as maxid from dual' " >> /apps/pentaho_nfs/installbase/input/poc/logs/IB_logs.log 2>&1
But, the command fails executing with a syntax error as the command seems to be split at the select query which is not being identified by sqoop cli. Whereas, when i run the same command directly on the cli it works like a charm.
Here, is my java code which creates a string array from the command and passes it to the ProcessBuilder object.
public void run() {
ProcessBuilder processbuilder = new ProcessBuilder(command);
try {
Process process = processbuilder.inheritIO().start();
process.waitFor();
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String readline;
while ((readline = reader.readLine()) != null) {
LOGGER.info(readline);
}
} catch (Exception e) {
LOGGER.error(e.getMessage());
}
}
static String[] command = null;
static String SQOOP_XXCCS_DS_SAHDR_CORE = ReadProperties.getInstance().getProperty("SQOOP_XXCCS_DS_SAHDR_CORE");
static String SQOOP_XXCCS_DS_CVDPRDLINE_DETAIL = ReadProperties.getInstance()
.getProperty("SQOOP_XXCCS_DS_CVDPRDLINE_DETAIL");
static String SQOOP_XXCCS_DS_INSTANCE_DETAIL = ReadProperties.getInstance()
.getProperty("SQOOP_XXCCS_DS_INSTANCE_DETAIL");
static String SQOOP_XXCCS_SCDC_PRODUCT_PROFILE = ReadProperties.getInstance()
.getProperty("SQOOP_XXCCS_SCDC_PRODUCT_PROFILE");
static String SQOOP_MTL_SYSTEM_ITEMS_B = ReadProperties.getInstance().getProperty("SQOOP_MTL_SYSTEM_ITEMS_B");
public static void main(String[] args) {
ThreadPoolExecutor executors = (ThreadPoolExecutor) Executors.newFixedThreadPool(5);
List<String> getlist = getList();
for (Iterator<String> itr = getlist.iterator(); itr.hasNext();) {
String element = (String) itr.next();
LOGGER.info(element);
command = element.split(" ");
LOGGER.info("Executing the command as below \n");
LOGGER.info(command);
ParallelExecutor pe = new ParallelExecutor(command);
executors.execute(pe);
}
executors.shutdown();
}
public static List<String> getList() {
List<String> commandlist = new ArrayList<String>();
commandlist.add(SQOOP_XXCCS_DS_SAHDR_CORE);
commandlist.add(SQOOP_XXCCS_DS_CVDPRDLINE_DETAIL);
commandlist.add(SQOOP_XXCCS_DS_INSTANCE_DETAIL);
commandlist.add(SQOOP_XXCCS_SCDC_PRODUCT_PROFILE);
commandlist.add(SQOOP_MTL_SYSTEM_ITEMS_B);
return commandlist;
}
Is there an elegant way i can pass the command to the processbuilder avoiding the syntactical errors. Need your valuable suggestions.
Stack Trace 1:
log4j:WARN No appenders could be found for logger (com.cisco.parallelexecutor.ParallelExecutor).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.io.IOException: Cannot run program "ssh hddev-c01-edge-02 "sqoop import -D mapred.child.java.opts='-Djava.security.egd=file:/dev/../dev/urandom' --connect 'jdbc:oracle:thin:#(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dbs-dev-vm-2009.cisco.com)(PORT=1533))(CONNECT_DATA=(SERVICE_NAME=SDADEV.CISCO.COM)(Server=Dedicated)))' --username XXCTS_NITA_U --password OBCK9I2Q --query "select CONTRACT_ID,CONTRACT_NUMBER,CONTRACT_STS_CODE,CONTRACT_STATUS,SERVICE_LINE_ID,SERVICE_LINE_NAME,SERVICE_LINE_STS_CODE,SERVICE_LINE_STATUS,SERVICE_LINE_NUMBER,HDR_CUST_ACCOUNT_ID,HDR_CUST_ACCOUNT_NUMBER,HDR_PARTY_ID,HDR_CUSTOMER_NAME,LINE_CUST_ACCOUNT_ID,LINE_CUST_ACCOUNT_NUMBER,LINE_PARTY_ID,LINE_CUSTOMER_NAME,BILL_TO_SITE_USE_ID,BILL_TO_SITE_USE_NAME,BILL_TO_SITE_USE_STS,BILL_TO_CUST_ACCT_ID,BILL_TO_CUST_ACCT_NUMBER,BILL_TO_CUST_ACCT_STS,BILL_TO_CUST_ACCT_SITE_ID,BILL_TO_CUST_ACCT_SITE_STS,BILL_TO_PARTY_ID,BILL_TO_PARTY_NUMBER,BILL_TO_CUSTOMER_NAME,BILL_TO_PARTY_STS,BILL_TO_PARTY_SITE_ID,BILL_TO_PARTY_SITE_STS,BILL_TO_LOCATION_ID,BILL_TO_COUNTRY,BILL_TO_STATE_PROV,BILL_TO_CITY,BILL_TO_POSTAL_CODE,COVERAGE_TEMPLATE_ID,COVERAGE_TEMPLATE_NAME,COVERAGE_TEMPLATE_DESC,COVERAGE_BEGIN_DATE,COVERAGE_END_DATE,SERVICE_SALES_REP_USR_ID,SERVICE_SALES_REP_USR_NAME,SERVICE_LINE_INVENTORY_ITEM_ID,BILLTO_CSC_ID,BILLTO_CSC_NAME,BILLTO_GU_ID,BILLTO_GU_NAME,BILL_TO_CUSTOMER_SALES_CHANNEL,CURRENCY_CODE,CONTRACT_LIST_AMT,CONTRACT_NET_AMT,USD_CONTRACT_LIST_AMT,USD_CONTRACT_NET_AMT,ACCOUNT_TEAM,RENEWAL_SSR,RENEWAL_TEAM,SALES_REGION,ADVANCE_CONTRACT_FLAG,EARLIEST_END_DATE,SERVICE_LINE_TERMINATED_DATE,SVC_LINE_TERMINATION_REASON,CONTRACT_TERMINATED_DATE,USD_CONV_RATE,PRICE_TYPE,NET_VALUE,COTERM_DATE,MARKETING_PROGRAM,MASTER_AGREEMENT_END_DATE,DISCOUNT_PERCENT,PRORATE_FLAG,RGP_ID,CREATION_DATE,CREATED_BY,LAST_UPDATE_DATE,LAST_UPDATED_BY,LAST_UPDATED_LOGIN,REQUEST_ID,PROGRAM_UPDATE_DATE,PROGRAM_APPLICATION_ID,PROGRAM_ID,ATTRIBUTE_CATEGORY,ATTRIBUTE1,ATTRIBUTE2,ATTRIBUTE3,ATTRIBUTE4,ATTRIBUTE5,ATTRIBUTE6,ATTRIBUTE7,ATTRIBUTE8,ATTRIBUTE9,ATTRIBUTE10,ATTRIBUTE11,ATTRIBUTE12,ATTRIBUTE13,ATTRIBUTE14,ATTRIBUTE15,ATTRIBUTE16,ATTRIBUTE17,ATTRIBUTE18,ATTRIBUTE19,ATTRIBUTE20,ACCOUNT_SSR,SHELL_CONTRACT_FLAG,CONTRACT_START_DATE,CONTRACT_END_DATE from XXCCS_DS_SAHDR_CORE where \$CONDITIONS " --split-by CONTRACT_NUMBER -m 4 --null-string '\\N' --null-non-string '\\N' --hive-delims-replacement '<EOL>' --boundary-query 'select (select min(CONTRACT_NUMBER) from XXCCS_DS_SAHDR_CORE) as minid ,(select max(CONTRACT_NUMBER) from XXCCS_DS_SAHDR_CORE) as maxid from dual' --target-dir /app/dev/SmartAnalytics/sqoop_temp//XXCCS_DS_SAHDR_CORE --hive-import --hive-overwrite --hive-table installbase.XXCCS_DS_SAHDR_CORE --map-column-hive CONTRACT_ID=BIGINT,SERVICE_LINE_ID=BIGINT,HDR_CUST_ACCOUNT_ID=BIGINT,HDR_PARTY_ID=BIGINT,LINE_CUST_ACCOUNT_ID=BIGINT,LINE_PARTY_ID=BIGINT,BILL_TO_SITE_USE_ID=BIGINT,BILL_TO_CUST_ACCT_ID=BIGINT,BILL_TO_CUST_ACCT_SITE_ID=BIGINT,BILL_TO_PARTY_ID=BIGINT,BILL_TO_PARTY_SITE_ID=BIGINT,BILL_TO_LOCATION_ID=BIGINT,COVERAGE_TEMPLATE_ID=BIGINT,COVERAGE_BEGIN_DATE=TIMESTAMP,COVERAGE_END_DATE=TIMESTAMP,SERVICE_LINE_INVENTORY_ITEM_ID=BIGINT,BILLTO_CSC_ID=BIGINT,BILLTO_GU_ID=BIGINT,CONTRACT_LIST_AMT=BIGINT,CONTRACT_NET_AMT=BIGINT,USD_CONTRACT_LIST_AMT=BIGINT,USD_CONTRACT_NET_AMT=BIGINT,EARLIEST_END_DATE=TIMESTAMP,SERVICE_LINE_TERMINATED_DATE=TIMESTAMP,CONTRACT_TERMINATED_DATE=TIMESTAMP,USD_CONV_RATE=BIGINT,NET_VALUE=BIGINT,MASTER_AGREEMENT_END_DATE=TIMESTAMP,DISCOUNT_PERCENT=BIGINT,PRORATE_FLAG=STRING,RGP_ID=BIGINT,CREATION_DATE=TIMESTAMP,CREATED_BY=BIGINT,LAST_UPDATE_DATE=TIMESTAMP,LAST_UPDATED_BY=BIGINT,LAST_UPDATED_LOGIN=BIGINT,REQUEST_ID=BIGINT,PROGRAM_UPDATE_DATE=TIMESTAMP,PROGRAM_APPLICATION_ID=BIGINT,PROGRAM_ID=BIGINT,ATTRIBUTE_CATEGORY=STRING,ATTRIBUTE1=STRING,ATTRIBUTE2=STRING,ATTRIBUTE3=STRING,ATTRIBUTE4=STRING,ATTRIBUTE5=STRING,ATTRIBUTE6=STRING,ATTRIBUTE7=STRING,ATTRIBUTE8=STRING,ATTRIBUTE9=STRING,ATTRIBUTE10=STRING,ATTRIBUTE11=STRING,ATTRIBUTE12=STRING,ATTRIBUTE13=STRING,ATTRIBUTE14=STRING,ATTRIBUTE15=STRING,ATTRIBUTE16=STRING,ATTRIBUTE17=STRING,ATTRIBUTE18=STRING,ATTRIBUTE19=STRING,ATTRIBUTE20=STRING,ACCOUNT_SSR=STRING,SHELL_CONTRACT_FLAG=STRING,CONTRACT_START_DATE=TIMESTAMP,CONTRACT_END_DATE=TIMESTAMP" >> /apps/pentaho_nfs/installbase/input/poc/logs/IB_logs.log 2>&1 ": error=36, File name too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
at com.cisco.parallelexecutor.ParallelExecutor.main(ParallelExecutor.java:35)
Caused by: java.io.IOException: error=36, File name too long
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:186)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
... 1 more
StackTrace 2:
Warning: /usr/lib/hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
--table or --query is required for import. (Or use sqoop import-all-tables.)
Try --help for usage instructions.
usage: sqoop import [GENERIC-ARGS] [TOOL-ARGS]
Common arguments:
--connect <jdbc-uri> Specify JDBC connect
string
--connection-manager <class-name> Specify connection manager
class name
--connection-param-file <properties-file> Specify connection
parameters file
--driver <class-name> Manually specify JDBC
driver class to use
--hadoop-home <hdir> Override
$HADOOP_MAPRED_HOME_ARG
--hadoop-mapred-home <dir> Override
$HADOOP_MAPRED_HOME_ARG
--help Print usage instructions
-P Read password from console
--password <password> Set authentication
password
--password-file <password-file> Set authentication
password file path
--username <username> Set authentication
username
--verbose Print more information
while working
Import control arguments:
--append Imports data
in append
mode
--as-avrodatafile Imports data
to Avro data
files
--as-sequencefile Imports data
to
SequenceFile
s
--as-textfile Imports data
as plain
text
(default)
--boundary-query <statement> Set boundary
query for
retrieving
max and min
value of the
primary key
--columns <col,col,col...> Columns to
import from
table
--compression-codec <codec> Compression
codec to use
for import
--delete-target-dir Imports data
in delete
mode
--direct Use direct
import fast
The issue is resolved after making the following changes in the code.
String[] command = { "ssh", "hddev-chost-01-01", "\"" + element + "\"" };
I am passing the command as a string array as per the ProcessBuilder documentation. Instead of sending the whole command as an array i am substituting the below command as as string "element".
String element = "sqoop import -D oraoop.disabled=true -D mapred.child.java.opts='\-Djava.security.egd=file:/dev/../dev/urandom' --connect 'jdbc:oracle:thin:#(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=maildude.com)(PORT=1533))(CONNECT_DATA=(SERVICE_NAME=SDA.COM)(Server=Dedicated)))' --username user --password pass --query \"select INVENTORY_ITEM_ID,ORGANIZATION_ID,LAST_UPDATE_DATE,LAST_UPDATED_BY,CREATION_DATE,CREATED_BY,LAST_UPDATE_LOGIN,SUMMARY_FLAG,ENABLED_FLAG,START_DATE_ACTIVE,END_DATE_ACTIVE,DESCRIPTION,BUYER_ID,ACCOUNTING_RULE_ID,INVOICING_RULE_ID,SEGMENT1,SEGMENT2,SEGMENT3,SEGMENT4,SEGMENT5,SEGMENT6,SEGMENT7,SEGMENT8,SEGMENT9,SEGMENT10,SEGMENT11,SEGMENT12,SEGMENT13,SEGMENT14,SEGMENT15,SEGMENT16,SEGMENT17,SEGMENT18,SEGMENT19,SEGMENT20,ATTRIBUTE_CATEGORY,ATTRIBUTE1,ATTRIBUTE2,ATTRIBUTE3,ATTRIBUTE4,ATTRIBUTE5,ATTRIBUTE6,ATTRIBUTE7,ATTRIBUTE8,ATTRIBUTE9,ATTRIBUTE10,ATTRIBUTE11,ATTRIBUTE12,ATTRIBUTE13,ATTRIBUTE14,ATTRIBUTE15,PURCHASING_ITEM_FLAG,SHIPPABLE_ITEM_FLAG,CUSTOMER_ORDER_FLAG,INTERNAL_ORDER_FLAG,SERVICE_ITEM_FLAG,INVENTORY_ITEM_FLAG,ENG_ITEM_FLAG,INVENTORY_ASSET_FLAG,PURCHASING_ENABLED_FLAG,CUSTOMER_ORDER_ENABLED_FLAG,INTERNAL_ORDER_ENABLED_FLAG,SO_TRANSACTIONS_FLAG,MTL_TRANSACTIONS_ENABLED_FLAG,STOCK_ENABLED_FLAG,BOM_ENABLED_FLAG,BUILD_IN_WIP_FLAG,REVISION_QTY_CONTROL_CODE,ITEM_CATALOG_GROUP_ID,CATALOG_STATUS_FLAG,RETURNABLE_FLAG,DEFAULT_SHIPPING_ORG,COLLATERAL_FLAG,TAXABLE_FLAG,QTY_RCV_EXCEPTION_CODE,ALLOW_ITEM_DESC_UPDATE_FLAG,INSPECTION_REQUIRED_FLAG,RECEIPT_REQUIRED_FLAG,MARKET_PRICE,HAZARD_CLASS_ID,RFQ_REQUIRED_FLAG,QTY_RCV_TOLERANCE,LIST_PRICE_PER_UNIT,UN_NUMBER_ID,PRICE_TOLERANCE_PERCENT,ASSET_CATEGORY_ID,ROUNDING_FACTOR,UNIT_OF_ISSUE,ENFORCE_SHIP_TO_LOCATION_CODE,ALLOW_SUBSTITUTE_RECEIPTS_FLAG,ALLOW_UNORDERED_RECEIPTS_FLAG,ALLOW_EXPRESS_DELIVERY_FLAG,DAYS_EARLY_RECEIPT_ALLOWED,DAYS_LATE_RECEIPT_ALLOWED,RECEIPT_DAYS_EXCEPTION_CODE,RECEIVING_ROUTING_ID,INVOICE_CLOSE_TOLERANCE,RECEIVE_CLOSE_TOLERANCE,AUTO_LOT_ALPHA_PREFIX,START_AUTO_LOT_NUMBER,LOT_CONTROL_CODE,SHELF_LIFE_CODE,SHELF_LIFE_DAYS,SERIAL_NUMBER_CONTROL_CODE,START_AUTO_SERIAL_NUMBER,AUTO_SERIAL_ALPHA_PREFIX,SOURCE_TYPE,SOURCE_ORGANIZATION_ID,SOURCE_SUBINVENTORY,EXPENSE_ACCOUNT,ENCUMBRANCE_ACCOUNT,RESTRICT_SUBINVENTORIES_CODE,UNIT_WEIGHT,WEIGHT_UOM_CODE,VOLUME_UOM_CODE,UNIT_VOLUME,RESTRICT_LOCATORS_CODE,LOCATION_CONTROL_CODE,SHRINKAGE_RATE,ACCEPTABLE_EARLY_DAYS,PLANNING_TIME_FENCE_CODE,DEMAND_TIME_FENCE_CODE,LEAD_TIME_LOT_SIZE,STD_LOT_SIZE,CUM_MANUFACTURING_LEAD_TIME,OVERRUN_PERCENTAGE,MRP_CALCULATE_ATP_FLAG,ACCEPTABLE_RATE_INCREASE,ACCEPTABLE_RATE_DECREASE,CUMULATIVE_TOTAL_LEAD_TIME,PLANNING_TIME_FENCE_DAYS,DEMAND_TIME_FENCE_DAYS,END_ASSEMBLY_PEGGING_FLAG,REPETITIVE_PLANNING_FLAG,PLANNING_EXCEPTION_SET,BOM_ITEM_TYPE,PICK_COMPONENTS_FLAG,REPLENISH_TO_ORDER_FLAG,BASE_ITEM_ID,ATP_COMPONENTS_FLAG,ATP_FLAG,FIXED_LEAD_TIME,VARIABLE_LEAD_TIME,WIP_SUPPLY_LOCATOR_ID,WIP_SUPPLY_TYPE,WIP_SUPPLY_SUBINVENTORY,PRIMARY_UOM_CODE,PRIMARY_UNIT_OF_MEASURE,ALLOWED_UNITS_LOOKUP_CODE,COST_OF_SALES_ACCOUNT,SALES_ACCOUNT,DEFAULT_INCLUDE_IN_ROLLUP_FLAG,INVENTORY_ITEM_STATUS_CODE,INVENTORY_PLANNING_CODE,PLANNER_CODE,PLANNING_MAKE_BUY_CODE,FIXED_LOT_MULTIPLIER,ROUNDING_CONTROL_TYPE,CARRYING_COST,POSTPROCESSING_LEAD_TIME,PREPROCESSING_LEAD_TIME,FULL_LEAD_TIME,ORDER_COST,MRP_SAFETY_STOCK_PERCENT,MRP_SAFETY_STOCK_CODE,MIN_MINMAX_QUANTITY,MAX_MINMAX_QUANTITY,MINIMUM_ORDER_QUANTITY,FIXED_ORDER_QUANTITY,FIXED_DAYS_SUPPLY,MAXIMUM_ORDER_QUANTITY,ATP_RULE_ID,PICKING_RULE_ID,RESERVABLE_TYPE,POSITIVE_MEASUREMENT_ERROR,NEGATIVE_MEASUREMENT_ERROR,ENGINEERING_ECN_CODE,ENGINEERING_ITEM_ID,ENGINEERING_DATE,SERVICE_STARTING_DELAY,VENDOR_WARRANTY_FLAG,SERVICEABLE_COMPONENT_FLAG,SERVICEABLE_PRODUCT_FLAG,BASE_WARRANTY_SERVICE_ID,PAYMENT_TERMS_ID,PREVENTIVE_MAINTENANCE_FLAG,PRIMARY_SPECIALIST_ID,SECONDARY_SPECIALIST_ID,SERVICEABLE_ITEM_CLASS_ID,TIME_BILLABLE_FLAG,MATERIAL_BILLABLE_FLAG,EXPENSE_BILLABLE_FLAG,PRORATE_SERVICE_FLAG,COVERAGE_SCHEDULE_ID,SERVICE_DURATION_PERIOD_CODE,SERVICE_DURATION,WARRANTY_VENDOR_ID,MAX_WARRANTY_AMOUNT,RESPONSE_TIME_PERIOD_CODE,RESPONSE_TIME_VALUE,NEW_REVISION_CODE,INVOICEABLE_ITEM_FLAG,TAX_CODE,INVOICE_ENABLED_FLAG,MUST_USE_APPROVED_VENDOR_FLAG,REQUEST_ID,PROGRAM_APPLICATION_ID,PROGRAM_ID,PROGRAM_UPDATE_DATE,OUTSIDE_OPERATION_FLAG,OUTSIDE_OPERATION_UOM_TYPE,SAFETY_STOCK_BUCKET_DAYS,AUTO_REDUCE_MPS,COSTING_ENABLED_FLAG,AUTO_CREATED_CONFIG_FLAG,CYCLE_COUNT_ENABLED_FLAG,ITEM_TYPE,MODEL_CONFIG_CLAUSE_NAME,SHIP_MODEL_COMPLETE_FLAG,MRP_PLANNING_CODE,RETURN_INSPECTION_REQUIREMENT,ATO_FORECAST_CONTROL,RELEASE_TIME_FENCE_CODE,RELEASE_TIME_FENCE_DAYS,CONTAINER_ITEM_FLAG,VEHICLE_ITEM_FLAG,MAXIMUM_LOAD_WEIGHT,MINIMUM_FILL_PERCENT,CONTAINER_TYPE_CODE,INTERNAL_VOLUME,WH_UPDATE_DATE,PRODUCT_FAMILY_ITEM_ID,GLOBAL_ATTRIBUTE_CATEGORY,GLOBAL_ATTRIBUTE1,GLOBAL_ATTRIBUTE2,GLOBAL_ATTRIBUTE3,GLOBAL_ATTRIBUTE4,GLOBAL_ATTRIBUTE5,GLOBAL_ATTRIBUTE6,GLOBAL_ATTRIBUTE7,GLOBAL_ATTRIBUTE8,GLOBAL_ATTRIBUTE9,GLOBAL_ATTRIBUTE10,PURCHASING_TAX_CODE,OVERCOMPLETION_TOLERANCE_TYPE,OVERCOMPLETION_TOLERANCE_VALUE,EFFECTIVITY_CONTROL,CHECK_SHORTAGES_FLAG,OVER_SHIPMENT_TOLERANCE,UNDER_SHIPMENT_TOLERANCE,OVER_RETURN_TOLERANCE,UNDER_RETURN_TOLERANCE,EQUIPMENT_TYPE,RECOVERED_PART_DISP_CODE,DEFECT_TRACKING_ON_FLAG,USAGE_ITEM_FLAG,EVENT_FLAG,ELECTRONIC_FLAG,DOWNLOADABLE_FLAG,VOL_DISCOUNT_EXEMPT_FLAG,COUPON_EXEMPT_FLAG,COMMS_NL_TRACKABLE_FLAG,ASSET_CREATION_CODE,COMMS_ACTIVATION_REQD_FLAG,ORDERABLE_ON_WEB_FLAG,BACK_ORDERABLE_FLAG,WEB_STATUS,INDIVISIBLE_FLAG,DIMENSION_UOM_CODE,UNIT_LENGTH,UNIT_WIDTH,UNIT_HEIGHT,BULK_PICKED_FLAG,LOT_STATUS_ENABLED,DEFAULT_LOT_STATUS_ID,SERIAL_STATUS_ENABLED,DEFAULT_SERIAL_STATUS_ID,LOT_SPLIT_ENABLED,LOT_MERGE_ENABLED,INVENTORY_CARRY_PENALTY,OPERATION_SLACK_PENALTY,FINANCING_ALLOWED_FLAG,EAM_ITEM_TYPE,EAM_ACTIVITY_TYPE_CODE,EAM_ACTIVITY_CAUSE_CODE,EAM_ACT_NOTIFICATION_FLAG,EAM_ACT_SHUTDOWN_STATUS,DUAL_UOM_CONTROL,SECONDARY_UOM_CODE,DUAL_UOM_DEVIATION_HIGH,DUAL_UOM_DEVIATION_LOW,CONTRACT_ITEM_TYPE_CODE,SUBSCRIPTION_DEPEND_FLAG,SERV_REQ_ENABLED_CODE,SERV_BILLING_ENABLED_FLAG,SERV_IMPORTANCE_LEVEL,PLANNED_INV_POINT_FLAG,LOT_TRANSLATE_ENABLED,DEFAULT_SO_SOURCE_TYPE,CREATE_SUPPLY_FLAG,SUBSTITUTION_WINDOW_CODE,SUBSTITUTION_WINDOW_DAYS,IB_ITEM_INSTANCE_CLASS,CONFIG_MODEL_TYPE,LOT_SUBSTITUTION_ENABLED,MINIMUM_LICENSE_QUANTITY,EAM_ACTIVITY_SOURCE_CODE,LIFECYCLE_ID,CURRENT_PHASE_ID,OBJECT_VERSION_NUMBER,CDB_DATA_SOURCE,CDB_ENQUEUE_TIME,CDB_DEQUEUE_TIME,CDB_ENQUEUE_SEQ from MTL_SYSTEM_ITEMS_B where \\\$CONDITIONS \" --target-dir /app/dev/SmartAnalytics/sqoop_temp//MTL_SYSTEM_ITEMS_B -m 4 --split-by INVENTORY_ITEM_ID --hive-import --hive-overwrite --hive-table installbase.MTL_SYSTEM_ITEMS_B --null-string '\\\\N' --null-non-string '\\\\N' --hive-delims-replacement '<EOL>' --map-column-hive INVENTORY_ITEM_ID=BIGINT,ORGANIZATION_ID=BIGINT,LAST_UPDATE_DATE=TIMESTAMP,LAST_UPDATED_BY=BIGINT,CREATION_DATE=TIMESTAMP,CREATED_BY=BIGINT,LAST_UPDATE_LOGIN=BIGINT,SUMMARY_FLAG=STRING,ENABLED_FLAG=STRING,START_DATE_ACTIVE=TIMESTAMP,END_DATE_ACTIVE=TIMESTAMP,DESCRIPTION=STRING,BUYER_ID=BIGINT,ACCOUNTING_RULE_ID=BIGINT,INVOICING_RULE_ID=BIGINT,SEGMENT1=STRING,SEGMENT2=STRING,SEGMENT3=STRING,SEGMENT4=STRING,SEGMENT5=STRING,SEGMENT6=STRING,SEGMENT7=STRING,SEGMENT8=STRING,SEGMENT9=STRING,SEGMENT10=STRING,SEGMENT11=STRING,SEGMENT12=STRING,SEGMENT13=STRING,SEGMENT14=STRING,SEGMENT15=STRING,SEGMENT16=STRING,SEGMENT17=STRING,SEGMENT18=STRING,SEGMENT19=STRING,SEGMENT20=STRING,ATTRIBUTE_CATEGORY=STRING,ATTRIBUTE1=STRING,ATTRIBUTE2=STRING,ATTRIBUTE3=STRING,ATTRIBUTE4=STRING,ATTRIBUTE5=STRING,ATTRIBUTE6=STRING,ATTRIBUTE7=STRING,ATTRIBUTE8=STRING,ATTRIBUTE9=STRING,ATTRIBUTE10=STRING,ATTRIBUTE11=STRING,ATTRIBUTE12=STRING,ATTRIBUTE13=STRING,ATTRIBUTE14=STRING,ATTRIBUTE15=STRING,PURCHASING_ITEM_FLAG=STRING,SHIPPABLE_ITEM_FLAG=STRING,CUSTOMER_ORDER_FLAG=STRING,INTERNAL_ORDER_FLAG=STRING,SERVICE_ITEM_FLAG=STRING,INVENTORY_ITEM_FLAG=STRING,ENG_ITEM_FLAG=STRING,INVENTORY_ASSET_FLAG=STRING,PURCHASING_ENABLED_FLAG=STRING,CUSTOMER_ORDER_ENABLED_FLAG=STRING,INTERNAL_ORDER_ENABLED_FLAG=STRING,SO_TRANSACTIONS_FLAG=STRING,MTL_TRANSACTIONS_ENABLED_FLAG=STRING,STOCK_ENABLED_FLAG=STRING,BOM_ENABLED_FLAG=STRING,BUILD_IN_WIP_FLAG=STRING,REVISION_QTY_CONTROL_CODE=BIGINT,ITEM_CATALOG_GROUP_ID=BIGINT,CATALOG_STATUS_FLAG=STRING,RETURNABLE_FLAG=STRING,DEFAULT_SHIPPING_ORG=BIGINT,COLLATERAL_FLAG=STRING,TAXABLE_FLAG=STRING,QTY_RCV_EXCEPTION_CODE=STRING,ALLOW_ITEM_DESC_UPDATE_FLAG=STRING,INSPECTION_REQUIRED_FLAG=STRING,RECEIPT_REQUIRED_FLAG=STRING,MARKET_PRICE=BIGINT,HAZARD_CLASS_ID=BIGINT,RFQ_REQUIRED_FLAG=STRING,QTY_RCV_TOLERANCE=BIGINT,LIST_PRICE_PER_UNIT=BIGINT,UN_NUMBER_ID=BIGINT,PRICE_TOLERANCE_PERCENT=BIGINT,ASSET_CATEGORY_ID=BIGINT,ROUNDING_FACTOR=BIGINT,UNIT_OF_ISSUE=STRING,ENFORCE_SHIP_TO_LOCATION_CODE=STRING,ALLOW_SUBSTITUTE_RECEIPTS_FLAG=STRING,ALLOW_UNORDERED_RECEIPTS_FLAG=STRING,ALLOW_EXPRESS_DELIVERY_FLAG=STRING,DAYS_EARLY_RECEIPT_ALLOWED=BIGINT,DAYS_LATE_RECEIPT_ALLOWED=BIGINT,RECEIPT_DAYS_EXCEPTION_CODE=STRING,RECEIVING_ROUTING_ID=BIGINT,INVOICE_CLOSE_TOLERANCE=BIGINT,RECEIVE_CLOSE_TOLERANCE=BIGINT,AUTO_LOT_ALPHA_PREFIX=STRING,START_AUTO_LOT_NUMBER=STRING,LOT_CONTROL_CODE=BIGINT,SHELF_LIFE_CODE=BIGINT,SHELF_LIFE_DAYS=BIGINT,SERIAL_NUMBER_CONTROL_CODE=BIGINT,START_AUTO_SERIAL_NUMBER=STRING,AUTO_SERIAL_ALPHA_PREFIX=STRING,SOURCE_TYPE=BIGINT,SOURCE_ORGANIZATION_ID=BIGINT,SOURCE_SUBINVENTORY=STRING,EXPENSE_ACCOUNT=BIGINT,ENCUMBRANCE_ACCOUNT=BIGINT,RESTRICT_SUBINVENTORIES_CODE=BIGINT,UNIT_WEIGHT=BIGINT,WEIGHT_UOM_CODE=STRING,VOLUME_UOM_CODE=STRING,UNIT_VOLUME=BIGINT,RESTRICT_LOCATORS_CODE=BIGINT,LOCATION_CONTROL_CODE=BIGINT,SHRINKAGE_RATE=BIGINT,ACCEPTABLE_EARLY_DAYS=BIGINT,PLANNING_TIME_FENCE_CODE=BIGINT,DEMAND_TIME_FENCE_CODE=BIGINT,LEAD_TIME_LOT_SIZE=BIGINT,STD_LOT_SIZE=BIGINT,CUM_MANUFACTURING_LEAD_TIME=BIGINT,OVERRUN_PERCENTAGE=BIGINT,MRP_CALCULATE_ATP_FLAG=STRING,ACCEPTABLE_RATE_INCREASE=BIGINT,ACCEPTABLE_RATE_DECREASE=BIGINT,CUMULATIVE_TOTAL_LEAD_TIME=BIGINT,PLANNING_TIME_FENCE_DAYS=BIGINT,DEMAND_TIME_FENCE_DAYS=BIGINT,END_ASSEMBLY_PEGGING_FLAG=STRING,REPETITIVE_PLANNING_FLAG=STRING,PLANNING_EXCEPTION_SET=STRING,BOM_ITEM_TYPE=BIGINT,PICK_COMPONENTS_FLAG=STRING,REPLENISH_TO_ORDER_FLAG=STRING,BASE_ITEM_ID=BIGINT,ATP_COMPONENTS_FLAG=STRING,ATP_FLAG=STRING,FIXED_LEAD_TIME=BIGINT,VARIABLE_LEAD_TIME=BIGINT,WIP_SUPPLY_LOCATOR_ID=BIGINT,WIP_SUPPLY_TYPE=BIGINT,WIP_SUPPLY_SUBINVENTORY=STRING,PRIMARY_UOM_CODE=STRING,PRIMARY_UNIT_OF_MEASURE=STRING,ALLOWED_UNITS_LOOKUP_CODE=BIGINT,COST_OF_SALES_ACCOUNT=BIGINT,SALES_ACCOUNT=BIGINT,DEFAULT_INCLUDE_IN_ROLLUP_FLAG=STRING,INVENTORY_ITEM_STATUS_CODE=STRING,INVENTORY_PLANNING_CODE=BIGINT,PLANNER_CODE=STRING,PLANNING_MAKE_BUY_CODE=BIGINT,FIXED_LOT_MULTIPLIER=BIGINT,ROUNDING_CONTROL_TYPE=BIGINT,CARRYING_COST=BIGINT,POSTPROCESSING_LEAD_TIME=BIGINT,PREPROCESSING_LEAD_TIME=BIGINT,FULL_LEAD_TIME=BIGINT,ORDER_COST=BIGINT,MRP_SAFETY_STOCK_PERCENT=BIGINT,MRP_SAFETY_STOCK_CODE=BIGINT,MIN_MINMAX_QUANTITY=BIGINT,MAX_MINMAX_QUANTITY=BIGINT,MINIMUM_ORDER_QUANTITY=BIGINT,FIXED_ORDER_QUANTITY=BIGINT,FIXED_DAYS_SUPPLY=BIGINT,MAXIMUM_ORDER_QUANTITY=BIGINT,ATP_RULE_ID=BIGINT,PICKING_RULE_ID=BIGINT,RESERVABLE_TYPE=BIGINT,POSITIVE_MEASUREMENT_ERROR=BIGINT,NEGATIVE_MEASUREMENT_ERROR=BIGINT,ENGINEERING_ECN_CODE=STRING,ENGINEERING_ITEM_ID=BIGINT,ENGINEERING_DATE=TIMESTAMP,SERVICE_STARTING_DELAY=BIGINT,VENDOR_WARRANTY_FLAG=STRING,SERVICEABLE_COMPONENT_FLAG=STRING,SERVICEABLE_PRODUCT_FLAG=STRING,BASE_WARRANTY_SERVICE_ID=BIGINT,PAYMENT_TERMS_ID=BIGINT,PREVENTIVE_MAINTENANCE_FLAG=STRING,PRIMARY_SPECIALIST_ID=BIGINT,SECONDARY_SPECIALIST_ID=BIGINT,SERVICEABLE_ITEM_CLASS_ID=BIGINT,TIME_BILLABLE_FLAG=STRING,MATERIAL_BILLABLE_FLAG=STRING,EXPENSE_BILLABLE_FLAG=STRING,PRORATE_SERVICE_FLAG=STRING,COVERAGE_SCHEDULE_ID=BIGINT,SERVICE_DURATION_PERIOD_CODE=STRING,SERVICE_DURATION=BIGINT,WARRANTY_VENDOR_ID=BIGINT,MAX_WARRANTY_AMOUNT=BIGINT,RESPONSE_TIME_PERIOD_CODE=STRING,RESPONSE_TIME_VALUE=BIGINT,NEW_REVISION_CODE=STRING,INVOICEABLE_ITEM_FLAG=STRING,TAX_CODE=STRING,INVOICE_ENABLED_FLAG=STRING,MUST_USE_APPROVED_VENDOR_FLAG=STRING,REQUEST_ID=BIGINT,PROGRAM_APPLICATION_ID=BIGINT,PROGRAM_ID=BIGINT,PROGRAM_UPDATE_DATE=TIMESTAMP,OUTSIDE_OPERATION_FLAG=STRING,OUTSIDE_OPERATION_UOM_TYPE=STRING,SAFETY_STOCK_BUCKET_DAYS=BIGINT,AUTO_REDUCE_MPS=BIGINT,COSTING_ENABLED_FLAG=STRING,AUTO_CREATED_CONFIG_FLAG=STRING,CYCLE_COUNT_ENABLED_FLAG=STRING,ITEM_TYPE=STRING,MODEL_CONFIG_CLAUSE_NAME=STRING,SHIP_MODEL_COMPLETE_FLAG=STRING,MRP_PLANNING_CODE=BIGINT,RETURN_INSPECTION_REQUIREMENT=BIGINT,ATO_FORECAST_CONTROL=BIGINT,RELEASE_TIME_FENCE_CODE=BIGINT,RELEASE_TIME_FENCE_DAYS=BIGINT,CONTAINER_ITEM_FLAG=STRING,VEHICLE_ITEM_FLAG=STRING,MAXIMUM_LOAD_WEIGHT=BIGINT,MINIMUM_FILL_PERCENT=BIGINT,CONTAINER_TYPE_CODE=STRING,INTERNAL_VOLUME=BIGINT,WH_UPDATE_DATE=TIMESTAMP,PRODUCT_FAMILY_ITEM_ID=BIGINT,GLOBAL_ATTRIBUTE_CATEGORY=STRING,GLOBAL_ATTRIBUTE1=STRING,GLOBAL_ATTRIBUTE2=STRING,GLOBAL_ATTRIBUTE3=STRING,GLOBAL_ATTRIBUTE4=STRING,GLOBAL_ATTRIBUTE5=STRING,GLOBAL_ATTRIBUTE6=STRING,GLOBAL_ATTRIBUTE7=STRING,GLOBAL_ATTRIBUTE8=STRING,GLOBAL_ATTRIBUTE9=STRING,GLOBAL_ATTRIBUTE10=STRING,PURCHASING_TAX_CODE=STRING,OVERCOMPLETION_TOLERANCE_TYPE=BIGINT,OVERCOMPLETION_TOLERANCE_VALUE=BIGINT,EFFECTIVITY_CONTROL=BIGINT,CHECK_SHORTAGES_FLAG=STRING,OVER_SHIPMENT_TOLERANCE=BIGINT,UNDER_SHIPMENT_TOLERANCE=BIGINT,OVER_RETURN_TOLERANCE=BIGINT,UNDER_RETURN_TOLERANCE=BIGINT,EQUIPMENT_TYPE=BIGINT,RECOVERED_PART_DISP_CODE=STRING,DEFECT_TRACKING_ON_FLAG=STRING,USAGE_ITEM_FLAG=STRING,EVENT_FLAG=STRING,ELECTRONIC_FLAG=STRING,DOWNLOADABLE_FLAG=STRING,VOL_DISCOUNT_EXEMPT_FLAG=STRING,COUPON_EXEMPT_FLAG=STRING,COMMS_NL_TRACKABLE_FLAG=STRING,ASSET_CREATION_CODE=STRING,COMMS_ACTIVATION_REQD_FLAG=STRING,ORDERABLE_ON_WEB_FLAG=STRING,BACK_ORDERABLE_FLAG=STRING,WEB_STATUS=STRING,INDIVISIBLE_FLAG=STRING,DIMENSION_UOM_CODE=STRING,UNIT_LENGTH=BIGINT,UNIT_WIDTH=BIGINT,UNIT_HEIGHT=BIGINT,BULK_PICKED_FLAG=STRING,LOT_STATUS_ENABLED=STRING,DEFAULT_LOT_STATUS_ID=BIGINT,SERIAL_STATUS_ENABLED=STRING,DEFAULT_SERIAL_STATUS_ID=BIGINT,LOT_SPLIT_ENABLED=STRING,LOT_MERGE_ENABLED=STRING,INVENTORY_CARRY_PENALTY=BIGINT,OPERATION_SLACK_PENALTY=BIGINT,FINANCING_ALLOWED_FLAG=STRING,EAM_ITEM_TYPE=BIGINT,EAM_ACTIVITY_TYPE_CODE=STRING,EAM_ACTIVITY_CAUSE_CODE=STRING,EAM_ACT_NOTIFICATION_FLAG=STRING,EAM_ACT_SHUTDOWN_STATUS=STRING,DUAL_UOM_CONTROL=BIGINT,SECONDARY_UOM_CODE=STRING,DUAL_UOM_DEVIATION_HIGH=BIGINT,DUAL_UOM_DEVIATION_LOW=BIGINT,CONTRACT_ITEM_TYPE_CODE=STRING,SUBSCRIPTION_DEPEND_FLAG=STRING,SERV_REQ_ENABLED_CODE=STRING,SERV_BILLING_ENABLED_FLAG=STRING,SERV_IMPORTANCE_LEVEL=BIGINT,PLANNED_INV_POINT_FLAG=STRING,LOT_TRANSLATE_ENABLED=STRING,DEFAULT_SO_SOURCE_TYPE=STRING,CREATE_SUPPLY_FLAG=STRING,SUBSTITUTION_WINDOW_CODE=BIGINT,SUBSTITUTION_WINDOW_DAYS=BIGINT,IB_ITEM_INSTANCE_CLASS=STRING,CONFIG_MODEL_TYPE=STRING,LOT_SUBSTITUTION_ENABLED=STRING,MINIMUM_LICENSE_QUANTITY=BIGINT,EAM_ACTIVITY_SOURCE_CODE=STRING,LIFECYCLE_ID=BIGINT,CURRENT_PHASE_ID=BIGINT,OBJECT_VERSION_NUMBER=BIGINT,CDB_DATA_SOURCE=STRING,CDB_ENQUEUE_TIME=TIMESTAMP,CDB_DEQUEUE_TIME=TIMESTAMP,CDB_ENQUEUE_SEQ=BIGINT --boundary-query 'select (select min(INVENTORY_ITEM_ID) from MTL_SYSTEM_ITEMS_B) as minid ,(select max(INVENTORY_ITEM_ID) from MTL_SYSTEM_ITEMS_B) as maxid from dual' " >> /apps/pentaho_nfs/installbase/input/poc/logs/IB_logs.log 2>&1"
public void run() {
ProcessBuilder processbuilder = new ProcessBuilder(command);
LOGGER.info(command);
try {
Process process = processbuilder.inheritIO().start();
System.out.println("inside process builder ");
process.waitFor();
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String readline;
while ((readline = reader.readLine()) != null) {
LOGGER.info(readline);
}
} catch (Exception e) {
LOGGER.error(e.getMessage());
}
}

Java maximum heap size on SGE cluster

I have a pipeline written in python calling some processes in Java. The pipeline runs with two possible modes, on local mode (on a single node) or on SGE cluster.
When I set the option to cluster mode, the error message in the logs are such this
Invalid maximum heap size: -Xmx4g -jar
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
In local mode, there is no error and no problems.
My question is what can cause such an error ?
My class to run jobs either locally or on cluster is as follow
class LocalJobManager(JobManager):
def __init__(self):
self.cmd_strs = []
def add_job(self, cmd, cmd_args, **kwargs):
cmd_str = ' '.join([cmd, ] + [str(x) for x in cmd_args])
self.cmd_strs.append(cmd_str)
def run_job(self, cmd, cmd_args, **kwargs):
cmd_str = ' '.join([cmd, ] + [str(x) for x in cmd_args])
self._run_cmd(cmd_str)
def wait(self):
for cmd_str in self.cmd_strs:
self._run_cmd(cmd_str)
def _run_cmd(self, cmd_str):
'''
Throw exception if run command fails
'''
process = subprocess.Popen(cmd_str, stdin=subprocess.PIPE, shell=True)
process.stdin.close()
sts = os.waitpid(process.pid, 0)
if sts[1] != 0:
raise Exception('Failed to run {0}\n'.format(cmd_str))
class ClusterJobManager(JobManager):
def __init__(self, log_dir=None):
import drmaa
self._drmaa = drmaa
self.log_dir = log_dir
if self.log_dir is not None:
make_directory(self.log_dir)
self.session = self._drmaa.Session()
self.session.initialize()
self.job_ids = Queue()
self._lock = threading.Lock()
def add_job(self, cmd, cmd_args, mem=4, max_mem=10, num_cpus=1):
job_id = self._run_job(cmd, cmd_args, mem, max_mem, num_cpus)
self.job_ids.put(job_id)
def run_job(self, cmd, cmd_args, mem=4, max_mem=10, num_cpus=1):
job_id = self._run_job(cmd, cmd_args, mem, max_mem, num_cpus)
self._check_exit_status(job_id)
def wait(self):
self._lock.acquire()
job_ids = []
while not self.job_ids.empty():
job_ids.append(self.job_ids.get())
self.session.synchronize(job_ids, self._drmaa.Session.TIMEOUT_WAIT_FOREVER, False)
self._lock.release()
for job_id in job_ids:
self._check_exit_status(job_id)
def close(self):
self.session.control(self._drmaa.Session.JOB_IDS_SESSION_ALL, self._drmaa.JobControlAction.TERMINATE)
self.session.exit()
def _run_job(self, cmd, cmd_args, mem, max_mem, num_cpus):
job_template = self._init_job_template(cmd, cmd_args, mem, max_mem, num_cpus)
job_id = self.session.runJob(job_template)
self.session.deleteJobTemplate(job_template)
return job_id
def _init_job_template(self, cmd, cmd_args, mem, max_mem, num_cpus):
job_template = self.session.createJobTemplate()
job_template.remoteCommand = cmd
job_template.args = [str(x) for x in cmd_args]
job_template.workingDirectory = os.getcwd()
if self.log_dir is not None:
job_template.errorPath = ':' + self.log_dir
job_template.outputPath = ':' + self.log_dir
job_template.nativeSpecification = '-l mem_free={mem}G,mem_token={mem}G,h_vmem={max_mem}G -V -w n -pe ncpus {num_cpus}'.format(**locals())
return job_template
def _check_exit_status(self, job_id):
return_value = self.session.wait(job_id, self._drmaa.Session.TIMEOUT_WAIT_FOREVER)
if return_value.exitStatus != 0:
raise Exception('Job {0} failed with exit status {1}.'.format(return_value.jobId,
return_value.exitStatus))
Usually the Could not create the Java Virtual Machine (as I am reading though some forum) is caused by syntax error, even though the command called is correct and works locally, besides the class to run jobs on the cluster show above, runs for everything except Java
Thanks
I've run into this on SGE. You may have a default hard memory limit set around 4GB and Java seems to use a bit more than that 4GB you are setting in the -Xmx4g parameter during initialization. Can you see if you administrator has set a hard memory limit? Typically you will set or override the default limit using:
qsub -l h_vmem=16G
Try giving far more memory than needed via that parameter and see if that solves it and then ratchet down the h_vmem as far down as it goes without crashing.

Scala - Fails to execute a process through terminal in a particular scenario

I'm using graphviz to generate graphs based on the messages passed in a scala program.
To invoke the graphviz application from inside the scala program, I'm using the exec() method (similar to Java). It successfully executed the command and created the graph when I used the below code snippet:
var cmd: String = "dot -Tpng Graph.dot -o Graph.png"
var run: Runtime = Runtime.getRuntime() ;
var pr: Process = run.exec(cmd) ;
However It fails to execute after changing the path of the input and output files (I just included a directory inside which the input file and output file resides as shown below)
def main(args: Array[String]): Unit = {
var DirectoryName: String = "Logs"
var GraphFileName: String = DirectoryName + File.separator + "Graph.dot"
val GraphFileObj: File = new File(GraphFileName)
// var cmd: String = "dot -Tpng Graph.dot -o Graph.png"
var cmd: String = "dot -Tpng \"" + GraphFileObj.getAbsolutePath + "\" -o \"" + DirectoryName + File.separator + "Graph.png\"" ;
println(cmd)
var run: Runtime = Runtime.getRuntime() ;
var pr: Process = run.exec(cmd) ;
}
The same command when executed through terminal gives proper output. Can you please help me to find what I'm missing?
exec is not a shell...e.g. quoting won't work as you expect, and thus your path (which may contain spaces, etc) will not be processed as you expect. The command will be broken apart using StringTokenizer, and your literal quotes will be...well..literal.
Use the form of exec that takes an array instead, so you can tokenize the command correctly.
val args = Array[String]("dot", "-Tpng", GraphFileObj.getAbsolutePath, ...);
run.exec(args)

PHP/Java bridge problem

I am using tomcat 6 on windows. Here is the code I am testing.
import java.io.ByteArrayOutputStream;
import java.io.Closeable;
import java.io.StringReader;
import javax.script.Invocable;
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
/**
* Create and run THREAD_COUNT PHP threads, concurrently accessing a
* shared resource.
*
* Create 5 script engines, passing each a shared resource allocated
* from Java. Each script engine has to implement Runnable.
*
* Java accesses the Runnable script engine using
* scriptEngine.getInterface() and calls thread.start() to invoke each
* PHP Runnable implementations concurrently.
*/
class PhpThreads {
public static final String runnable = new String("<?php\n" +
"function run() {\n" +
" $out = java_context()->getAttribute('sharedResource', 100);\n" +
" $nr = (string)java_context()->getAttribute('nr', 100);\n" +
" echo \"started thread: $nr\n\";\n" +
" for($i=0; $i<100; $i++) {\n" +
" $out->write(ord($nr));\n" +
" java('java.lang.Thread')->sleep(1);\n" +
" }\n" +
"}\n" +
"?>\n");
static final int THREAD_COUNT = 5;
public static void main(String[] args) throws Exception {
ScriptEngineManager manager = new ScriptEngineManager();
Thread threads[] = new Thread[THREAD_COUNT];
ScriptEngine engines[] = new ScriptEngine[THREAD_COUNT];
ByteArrayOutputStream sharedResource = new ByteArrayOutputStream();
StringReader runnableReader = new StringReader(runnable);
// create THREAD_COUNT PHP threads
for (int i=0; i<THREAD_COUNT; i++) {
engines[i] = manager.getEngineByName("php-invocable");
if (engines[i] == null)
throw new NullPointerException ("php script engine not found");
engines[i].put("nr", new Integer(i+1));
engines[i].put("sharedResource", sharedResource);
engines[i].eval(runnableReader);
runnableReader.reset();
// cast the whole script to Runnable; note also getInterface(specificClosure, type)
Runnable r = (Runnable) ((Invocable)engines[i]).getInterface(Runnable.class);
threads[i] = new Thread(r);
}
// run the THREAD_COUNT PHP threads
for (int i=0; i<THREAD_COUNT; i++) {
threads[i].start();
}
// wait for the THREAD_COUNT PHP threads to finish
for (int i=0; i<THREAD_COUNT; i++) {
threads[i].join();
((Closeable)engines[i]).close();
}
// print the output generated by the THREAD_COUNT concurrent threads
String result = sharedResource.toString();
System.out.println(result);
// Check result
Object res=manager.getEngineByName("php").eval(
"<?php " +
"exit((int)('10011002100310041005'!=" +
"#system(\"echo -n "+result+"|sed 's/./&\\\n/g'|sort|uniq -c|tr -d ' \\\n'\")));" +
"?>");
System.exit(((Number)res).intValue());
}
}
I have added all the libraries. When I run the file I get the following error -
run:
Exception in thread "main" javax.script.ScriptException: java.io.IOException: Cannot run program "php-cgi": CreateProcess error=2, The system cannot find the file specified
at php.java.script.InvocablePhpScriptEngine.eval(InvocablePhpScriptEngine.java:209)
at php.java.script.SimplePhpScriptEngine.eval(SimplePhpScriptEngine.java:178)
at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:232)
at PhpThreads.main(NewClass.java:53)
Caused by: java.io.IOException: Cannot run program "php-cgi": CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
at java.lang.Runtime.exec(Runtime.java:593)
at php.java.bridge.Util$Process.start(Util.java:1064)
at php.java.bridge.Util$ProcessWithErrorHandler.start(Util.java:1166)
at php.java.bridge.Util$ProcessWithErrorHandler.start(Util.java:1217)
at php.java.script.CGIRunner.doRun(CGIRunner.java:126)
at php.java.script.HttpProxy.doRun(HttpProxy.java:63)
at php.java.script.CGIRunner.run(CGIRunner.java:111)
at php.java.bridge.ThreadPool$Delegate.run(ThreadPool.java:60)
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.<init>(ProcessImpl.java:81)
at java.lang.ProcessImpl.start(ProcessImpl.java:30)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
... 8 more
What am I missing?
Just add this to your command line:
-Dphp.java.bridge.php_exec=/usr/bin/php
Problem solved!
copy the correct VERSION of the PHP of php5ts.dll and php-cgi.exe to "WEB-INF\cgi\amd64-windows" directory. then restart Tomcat. good luck.
...php-cgi...The system cannot find the file specified
I'm guessing that manager.getEngineByName("php-invocable") should return a wrapper around a system call to run PHP - but that wrapper doesn't know where to find the PHP executable.
A quick glance at the website for the PHP/Java bridge, and I infer that the path is hard coded in the Java - "For further information please see the INSTALL.J2EE file from the documentation download"
The Javadoc is decidedly vague on the topic.
You need to specifically make the -cgi version of PHP at compile time, assuming you've done that, and it is called php-cgi, then as a quick hack you could perpper your filesystem with links named "php-cgi" (its probably expected to be in /bin, /usr/bin/, /usr/local/bin, or the Java may be smart enough to check $PATH)
C.
when you get error like
Fatal Error: Failed to start PHP ["php-cgi", "-v"], reason: java.io.IOException:
Cannot run program ""php-cgi"" (in directory "C:\Documents and Settings\Adminis
trator"): CreateProcess error=2, The system cannot find the file specified
Could not start FCGI server: java.io.IOException: PHP not found. Please install
php-cgi. PHP test command was: [php-cgi, -v]
php.java.bridge.http.FCGIConnectException: Could not connect to server
at php.java.bridge.http.NPChannelFactory.test(NPChannel`enter code here`Factory.java:64)
at php.java.bridge.http.FCGIConnectionPool.<init>(FCGIConnectionPool.jav
a:175)
at php.java.bridge.http.FCGIConnectionPool.<init>(FCGIConnectionPool.jav
a:189)
at php.java.servlet.ContextLoaderListener.createConnectionPool(ContextLo
aderListener.java:541)
at php.java.servlet.ContextLoaderListener.contextInitialized(ContextLoad
erListener.java:185)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContex
t.java:4135)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4
630)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase
.java:791)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:77
1)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:546)
at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.jav
a:1041)
at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.j
ava:964)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502
)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1277)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java
:321)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(Lifecycl
eSupport.java:119)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:785)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:445
)
at org.apache.catalina.core.StandardService.start(StandardService.java:5
19)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:710
)
at org.apache.catalina.startup.Catalina.start(Catalina.java:581)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: java.io.IOException: File \\.\pipe\C:\Documents and Settings\Administ
rator\Desktop\softwares\apache-tomcat-6.0.29\temp\JavaBridge3144995283109409611.
socket not writable
at php.java.bridge.http.FCGIConnectException.<init>(FCGIConnectException
.java:37)
... 29 more
Caused by: java.io.IOException: PHP not found. Please install php-cgi. PHP test
command was: [php-cgi, -v]
at php.java.bridge.Util$Process.start(Util.java:1145)
at php.java.servlet.fastcgi.FCGIProcess.start(FCGIProcess.java:68)
at php.java.bridge.http.NPChannelFactory.doBind(NPChannelFactory.java:94
)
at php.java.bridge.http.FCGIConnectionFactory.runFcgi(FCGIConnectionFact
ory.java:88)
at php.java.bridge.http.FCGIConnectionFactory$1.run(FCGIConnectionFactor
y.java:109)
with the JavaBridge.war deployment in (windows, tomcat)
please specify the path for the php installation in your environment variable.

Categories