i am running this import command to import E_INFO table from MySQL databases to HDFS using sqoop. the job is failing with below given error.
sqoop import --connect jdbc:mysql://localhost/HADOOP_BANK_DATA_POC --username root --password xxxxxxx --table E_INFO
however a blank file is getting created without any records in hdfs as E_INFO.
your help here is highly appreciated.
h-user#h-primary:~/sqoop-1.4.7/bin$ sqoop import --connect jdbc:mysql://localhost/HADOOP_BANK_DATA_POC --username root --password xxxxxxx --table E_INFO
Warning: /home/h-user/sqoop-1.4.7/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /home/h-user/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/h-user/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/h-user/sqoop-1.4.7/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
/usr/local/hadoop/libexec/hadoop-functions.sh: line 2366: HADOOP_ORG.APACHE.SQOOP.SQOOP_USER: invalid variable name
/usr/local/hadoop/libexec/hadoop-functions.sh: line 2461: HADOOP_ORG.APACHE.SQOOP.SQOOP_OPTS: invalid variable name
2022-12-24 21:33:59,888 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
2022-12-24 21:34:00,034 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
2022-12-24 21:34:00,355 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
2022-12-24 21:34:00,356 INFO tool.CodeGenTool: Beginning code generation
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
2022-12-24 21:34:02,279 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `E_INFO` AS t LIMIT 1
2022-12-24 21:34:02,678 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `E_INFO` AS t LIMIT 1
2022-12-24 21:34:02,693 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/local/hadoop
Note: /tmp/sqoop-h-user/compile/e0e749532f0eaa6daaca522c9759658a/E_INFO.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
2022-12-24 21:34:07,748 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-h-user/compile/e0e749532f0eaa6daaca522c9759658a/E_INFO.jar
2022-12-24 21:34:07,798 WARN manager.MySQLManager: It looks like you are importing from mysql.
2022-12-24 21:34:07,798 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
2022-12-24 21:34:07,798 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
2022-12-24 21:34:07,798 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
2022-12-24 21:34:07,830 INFO mapreduce.ImportJobBase: Beginning import of E_INFO
2022-12-24 21:34:07,833 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2022-12-24 21:34:08,191 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
2022-12-24 21:34:09,386 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2022-12-24 21:34:09,626 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
2022-12-24 21:34:10,080 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2022-12-24 21:34:10,080 INFO impl.MetricsSystemImpl: JobTracker metrics system started
2022-12-24 21:34:10,722 INFO db.DBInputFormat: Using read commited transaction isolation
2022-12-24 21:34:10,793 INFO mapreduce.JobSubmitter: number of splits:1
2022-12-24 21:34:11,333 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local377089840_0001
2022-12-24 21:34:11,334 INFO mapreduce.JobSubmitter: Executing with tokens: []
2022-12-24 21:34:12,059 INFO mapred.LocalDistributedCacheManager: Creating symlink: /tmp/hadoop-h-user/mapred/local/job_local377089840_0001_7cb16b63-36d5-4781-94e7-4bd839e2ca49/libjars <- /home/h-user/sqoop-1.4.7/bin/libjars/*
2022-12-24 21:34:12,098 WARN fs.FileUtil: Command 'ln -s /tmp/hadoop-h-user/mapred/local/job_local377089840_0001_7cb16b63-36d5-4781-94e7-4bd839e2ca49/libjars /home/h-user/sqoop-1.4.7/bin/libjars/*' failed 1 with: ln: failed to create symbolic link '/home/h-user/sqoop-1.4.7/bin/libjars/*': No such file or directory
2022-12-24 21:34:12,098 WARN mapred.LocalDistributedCacheManager: Failed to create symlink: /tmp/hadoop-h-user/mapred/local/job_local377089840_0001_7cb16b63-36d5-4781-94e7-4bd839e2ca49/libjars <- /home/h-user/sqoop-1.4.7/bin/libjars/*
2022-12-24 21:34:12,099 INFO mapred.LocalDistributedCacheManager: Localized file:/tmp/hadoop/mapred/staging/h-user377089840/.staging/job_local377089840_0001/libjars as file:/tmp/hadoop-h-user/mapred/local/job_local377089840_0001_7cb16b63-36d5-4781-94e7-4bd839e2ca49/libjars
2022-12-24 21:34:12,256 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
2022-12-24 21:34:12,261 INFO mapred.LocalJobRunner: OutputCommitter set in config null
2022-12-24 21:34:12,264 INFO mapreduce.Job: Running job: job_local377089840_0001
2022-12-24 21:34:12,352 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-12-24 21:34:12,353 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-12-24 21:34:12,362 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2022-12-24 21:34:12,743 INFO mapred.LocalJobRunner: Waiting for map tasks
2022-12-24 21:34:12,745 INFO mapred.LocalJobRunner: Starting task: attempt_local377089840_0001_m_000000_0
2022-12-24 21:34:12,831 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2022-12-24 21:34:12,841 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2022-12-24 21:34:12,918 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2022-12-24 21:34:13,007 INFO db.DBInputFormat: Using read commited transaction isolation
2022-12-24 21:34:13,012 INFO mapred.MapTask: Processing split: 1=1 AND 1=1
2022-12-24 21:34:13,025 INFO mapred.LocalJobRunner: map task executor complete.
2022-12-24 21:34:13,134 WARN mapred.LocalJobRunner: job_local377089840_0001
java.lang.Exception: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class E_INFO not found
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class E_INFO not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2638)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getInputClass(DBConfiguration.java:403)
at org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.createDBRecordReader(DataDrivenDBInputFormat.java:270)
at org.apache.sqoop.mapreduce.db.DBInputFormat.createRecordReader(DBInputFormat.java:266)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:527)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassNotFoundException: Class E_INFO not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2542)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2636)
... 12 more
2022-12-24 21:34:13,345 INFO mapreduce.Job: Job job_local377089840_0001 running in uber mode : false
2022-12-24 21:34:13,351 INFO mapreduce.Job: map 0% reduce 0%
2022-12-24 21:34:13,354 INFO mapreduce.Job: Job job_local377089840_0001 failed with state FAILED due to: NA
2022-12-24 21:34:13,389 INFO mapreduce.Job: Counters: 0
2022-12-24 21:34:13,396 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
2022-12-24 21:34:13,402 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 3.998 seconds (0 bytes/sec)
2022-12-24 21:34:13,404 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2022-12-24 21:34:13,404 INFO mapreduce.ImportJobBase: Retrieved 0 records.
2022-12-24 21:34:13,404 ERROR tool.ImportTool: Import failed: Import job failed!
Related
I'm using HADOOP on a VM. When I try to run a jar, the execution stops because is unable to find the file resource-type.xml.
How can I solve this? Thank you.
gaia#gaia-virtual-machine:~/hadoop-3.3.2$ bin/hadoop jar erasmus-0.0.1-SNAPSHOT.jar erasmus.MaxPartecipants input output
2022-05-09 10:27:30,069 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032
2022-05-09 10:27:30,439 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2022-05-09 10:27:30,457 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/gaia/.staging/job_1652083747035_0002
2022-05-09 10:27:30,673 INFO input.FileInputFormat: Total input files to process : 1
2022-05-09 10:27:31,158 INFO mapreduce.JobSubmitter: number of splits:1
2022-05-09 10:27:31,268 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1652083747035_0002
2022-05-09 10:27:31,268 INFO mapreduce.JobSubmitter: Executing with tokens: []
2022-05-09 10:27:31,439 INFO conf.Configuration: resource-types.xml not found
2022-05-09 10:27:31,440 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2022-05-09 10:27:31,516 INFO impl.YarnClientImpl: Submitted application application_1652083747035_0002
2022-05-09 10:27:31,553 INFO mapreduce.Job: The url to track the job: http://gaia-virtual-machine:8088/proxy/application_1652083747035_0002/
2022-05-09 10:27:31,554 INFO mapreduce.Job: Running job: job_1652083747035_0002
The following is the output of the jps command:
gaia#gaia-virtual-machine:~/hadoop-3.3.2$ jps
14998 SecondaryNameNode
14648 NameNode
14779 DataNode
17836 Jps
16780 ResourceManager
In the yarn web UI it says that: Total Resource Preempted: <memory:0, vCores:0>
And in the node sections it says that there are 0 active nodes
I was running this sqoop query
sqoop export -D oraoop.timestamp.string=false --connect jdbc:oracle:thin:#127.0.0.1:1521:XE --username root --password manager --table person_temp4 --columns "id,first_name,last_name,email,tim" --export-dir /sqoop/person_data/avro3/ -m 4
For this query Oraoop was invoked, which is strange as oraoop doesn`t support export or eval jobs
The output is :
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
17/04/13 04:01:07 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4.2.1.1.0-385
17/04/13 04:01:07 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
17/04/13 04:01:07 INFO manager.SqlManager: Using default fetchSize of 1000
17/04/13 04:01:08 INFO oraoop.OraOopOracleQueries: Current schema is: ROOT
17/04/13 04:01:08 INFO oraoop.OraOopManagerFactory:
***********************************************************************
*** Using Quest® Data Connector for Oracle and Hadoop 1.6.0-cdh4-20 ***
*** Copyright 2012 Quest Software, Inc. ***
*** ALL RIGHTS RESERVED. ***
***********************************************************************
17/04/13 04:01:08 INFO oraoop.OraOopManagerFactory: Oracle Database version: Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production
17/04/13 04:01:08 INFO oraoop.OraOopManagerFactory: This Oracle database is not a RAC.
17/04/13 04:01:08 INFO Configuration.deprecation: mapred.map.max.attempts is deprecated. Instead, use mapreduce.map.maxattempts
17/04/13 04:01:08 INFO tool.CodeGenTool: Beginning code generation
17/04/13 04:01:09 INFO manager.SqlManager: Executing SQL statement: SELECT "ID","FIRST_NAME","LAST_NAME","EMAIL","TIM" FROM person_temp4 WHERE 0=1
17/04/13 04:01:09 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
Note: /tmp/sqoop-root/compile/b4748ab65cf1d407986f9bee683d82db/person_temp4.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/04/13 04:01:11 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/b4748ab65cf1d407986f9bee683d82db/person_temp4.jar
17/04/13 04:01:11 INFO mapreduce.ExportJobBase: Beginning export of person_temp4
17/04/13 04:01:11 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
17/04/13 04:01:13 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
17/04/13 04:01:13 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
17/04/13 04:01:13 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
17/04/13 04:01:13 INFO client.RMProxy: Connecting to ResourceManager at sandbox.hortonworks.com/10.0.2.15:8050
17/04/13 04:01:14 INFO input.FileInputFormat: Total input paths to process : 1
17/04/13 04:01:14 INFO mapreduce.JobSubmitter: number of splits:1
17/04/13 04:01:14 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
17/04/13 04:01:14 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1492080500165_0005
17/04/13 04:01:15 INFO impl.YarnClientImpl: Submitted application application_1492080500165_0005
17/04/13 04:01:15 INFO mapreduce.Job: The url to track the job: http://sandbox.hortonworks.com:8088/proxy/application_1492080500165_0005/
17/04/13 04:01:15 INFO mapreduce.Job: Running job: job_1492080500165_0005
17/04/13 04:01:23 INFO mapreduce.Job: Job job_1492080500165_0005 running in uber mode : false
17/04/13 04:01:23 INFO mapreduce.Job: map 0% reduce 0%
17/04/13 04:01:32 INFO mapreduce.Job: Task Id : attempt_1492080500165_0005_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: No such field: TIM
at person_temp4.setField(person_temp4.java:337)
at org.apache.sqoop.mapreduce.AvroExportMapper.toSqoopRecord(AvroExportMapper.java:120)
at org.apache.sqoop.mapreduce.AvroExportMapper.map(AvroExportMapper.java:104)
at org.apache.sqoop.mapreduce.AvroExportMapper.map(AvroExportMapper.java:49)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
17/04/13 04:01:38 INFO mapreduce.Job: Task Id : attempt_1492080500165_0005_m_000000_1, Status : FAILED
Error: java.lang.RuntimeException: No such field: TIM
at person_temp4.setField(person_temp4.java:337)
at org.apache.sqoop.mapreduce.AvroExportMapper.toSqoopRecord(AvroExportMapper.java:120)
at org.apache.sqoop.mapreduce.AvroExportMapper.map(AvroExportMapper.java:104)
at org.apache.sqoop.mapreduce.AvroExportMapper.map(AvroExportMapper.java:49)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
17/04/13 04:01:44 INFO mapreduce.Job: Task Id : attempt_1492080500165_0005_m_000000_2, Status : FAILED
Error: java.lang.RuntimeException: No such field: TIM
at person_temp4.setField(person_temp4.java:337)
at org.apache.sqoop.mapreduce.AvroExportMapper.toSqoopRecord(AvroExportMapper.java:120)
at org.apache.sqoop.mapreduce.AvroExportMapper.map(AvroExportMapper.java:104)
at org.apache.sqoop.mapreduce.AvroExportMapper.map(AvroExportMapper.java:49)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
17/04/13 04:01:56 INFO mapreduce.Job: map 100% reduce 0%
17/04/13 04:01:57 INFO mapreduce.Job: Job job_1492080500165_0005 failed with state FAILED due to: Task failed task_1492080500165_0005_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
17/04/13 04:01:57 INFO mapreduce.Job: Counters: 9
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=3
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=25923
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=25923
Total vcore-seconds taken by all map tasks=25923
Total megabyte-seconds taken by all map tasks=6480750
17/04/13 04:01:57 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
17/04/13 04:01:57 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 44.3748 seconds (0 bytes/sec)
17/04/13 04:01:57 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
17/04/13 04:01:57 INFO mapreduce.ExportJobBase: Exported 0 records.
17/04/13 04:01:57 ERROR tool.ExportTool: Error during export: Export job failed!
Why is Oraoop being invoked for export jobs?
I am using HDP 2.1 and have installed Oraoop seperately.
15/06/11 10:31:51 INFO mapreduce.Job: map 100% reduce 0%
I am trying to run open source kNN join MapReduce hbrj algorithm on a Hadoop 2.6.0 for single node cluster - pseudo-distributed operation installed on my laptop (OSX). (The source can be found here: http://www.cs.utah.edu/~lifeifei/knnj/). This algorithm is comprised of two MapReduce phases where the second phase uses the first phase's output files as its input. The first phase maps and reduces successfully - I can also look into the output files and everything seems right. However, when running the second phase the job is said to finish successfully even though it never reduces or even enters that stage I believe.
Here is what gets printed as I run phase 2 (I am including everything in hopes that it can be useful)
2015-06-11 10:31:47.526 java[3918:305930] Unable to load realm info from SCDynamicStore
15/06/11 10:31:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/06/11 10:31:49 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
15/06/11 10:31:49 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
15/06/11 10:31:49 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
15/06/11 10:31:49 INFO mapred.FileInputFormat: Total input paths to process : 64
15/06/11 10:31:49 INFO mapreduce.JobSubmitter: number of splits:64
15/06/11 10:31:50 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1089761712_0001
15/06/11 10:31:50 INFO mapred.LocalJobRunner: OutputCommitter set in config null
15/06/11 10:31:50 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
15/06/11 10:31:50 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
15/06/11 10:31:50 INFO mapreduce.Job: Running job: job_local1089761712_0001
15/06/11 10:31:50 INFO mapred.LocalJobRunner: Waiting for map tasks
15/06/11 10:31:50 INFO mapred.LocalJobRunner: Starting task: attempt_local1089761712_0001_m_000000_0
15/06/11 10:31:50 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/06/11 10:31:50 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/06/11 10:31:50 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/sasha/hbrj/output/part-00042:0+872
15/06/11 10:31:50 INFO mapred.MapTask: numReduceTasks: 0
15/06/11 10:31:50 INFO mapred.LocalJobRunner:
15/06/11 10:31:50 INFO mapred.Task: Task:attempt_local1089761712_0001_m_000000_0 is done. And is in the process of committing
15/06/11 10:31:50 INFO mapred.LocalJobRunner:
15/06/11 10:31:50 INFO mapred.Task: Task attempt_local1089761712_0001_m_000000_0 is allowed to commit now
15/06/11 10:31:50 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1089761712_0001_m_000000_0' to hdfs://localhost:9000/user/sasha/hbrj/output2/_temporary/0/task_local1089761712_0001_m_000000
15/06/11 10:31:50 INFO mapred.MapTask: numReduceTasks: 0
15/06/11 10:31:50 INFO mapred.LocalJobRunner:
15/06/11 10:31:50 INFO mapred.Task: Task:attempt_local1089761712_0001_m_000000_0 is done. And is in the process of committing
15/06/11 10:31:50 INFO mapred.LocalJobRunner:
15/06/11 10:31:50 INFO mapred.Task: Task attempt_local1089761712_0001_m_000000_0 is allowed to commit now
15/06/11 10:31:50 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1089761712_0001_m_000000_0' to hdfs://localhost:9000/user/sasha/hbrj/output2/_temporary/0/task_local1089761712_0001_m_000000
15/06/11 10:31:50 INFO mapred.LocalJobRunner: hdfs://localhost:9000/user/sasha/hbrj/output/part-00042:0+872
15/06/11 10:31:50 INFO mapred.Task: Task 'attempt_local1089761712_0001_m_000000_0' done.
continues in this fashion until...
15/06/11 10:31:51 INFO mapred.LocalJobRunner: Finishing task: attempt_local1089761712_0001_m_000012_0
15/06/11 10:31:51 INFO mapred.LocalJobRunner: Starting task: attempt_local1089761712_0001_m_000013_0
15/06/11 10:31:51 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/06/11 10:31:51 INFO mapred.Task: Using ResourceCalculatorProcessTree : null
15/06/11 10:31:51 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/sasha/hbrj/output/part-00015:0+646
15/06/11 10:31:51 INFO mapred.MapTask: numReduceTasks: 0
15/06/11 10:31:51 INFO mapreduce.Job: Job job_local1089761712_0001 running in uber mode : false
15/06/11 10:31:51 INFO mapreduce.Job: map 100% reduce 0%
15/06/11 10:31:51 INFO mapred.LocalJobRunner:
15/06/11 10:31:51 INFO mapred.Task: Task:attempt_local1089761712_0001_m_000013_0 is done. And is in the process of committing
15/06/11 10:31:51 INFO mapred.LocalJobRunner:
15/06/11 10:31:51 INFO mapred.Task: Task attempt_local1089761712_0001_m_000013_0 is allowed to commit now
15/06/11 10:31:51 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1089761712_0001_m_000013_0' to hdfs://localhost:9000/user/sasha/hbrj/output2/_temporary/0/task_local1089761712_0001_m_000013
15/06/11 10:31:51 INFO mapred.LocalJobRunner: hdfs://localhost:9000/user/sasha/hbrj/output/part-00015:0+646
15/06/11 10:31:51 INFO mapred.Task: Task 'attempt_local1089761712_0001_m_000013_0' done.
15/06/11 10:31:51 INFO mapred.LocalJobRunner: Finishing task: attempt_local1089761712_0001_m_000013_0
15/06/11 10:31:51 INFO mapred.LocalJobRunner: Starting task: attempt_local1089761712_0001_m_000014_0
Starting task ... Finishing task
repeats in the manner as seen below (which is the last task) and the job is said to complete successfully:
15/06/11 10:31:53 INFO mapred.MapTask: numReduceTasks: 0
15/06/11 10:31:53 INFO mapred.LocalJobRunner:
15/06/11 10:31:53 INFO mapred.Task: Task:attempt_local1089761712_0001_m_000063_0 is done. And is in the process of committing
15/06/11 10:31:53 INFO mapred.LocalJobRunner:
15/06/11 10:31:53 INFO mapred.Task: Task attempt_local1089761712_0001_m_000063_0 is allowed to commit now
15/06/11 10:31:53 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1089761712_0001_m_000063_0' to hdfs://localhost:9000/user/sasha/hbrj/output2/_temporary/0/task_local1089761712_0001_m_000063
15/06/11 10:31:53 INFO mapred.LocalJobRunner: hdfs://localhost:9000/user/sasha/hbrj/output/part-00004:0+178
15/06/11 10:31:53 INFO mapred.Task: Task 'attempt_local1089761712_0001_m_000063_0' done.
15/06/11 10:31:53 INFO mapred.LocalJobRunner: Finishing task: attempt_local1089761712_0001_m_000063_0
15/06/11 10:31:53 INFO mapred.LocalJobRunner: map task executor complete.
15/06/11 10:31:54 INFO mapreduce.Job: Job job_local1089761712_0001 completed successfully
15/06/11 10:31:54 INFO mapreduce.Job: Counters: 20
File System Counters
FILE: Number of bytes read=96487226
FILE: Number of bytes written=106993472
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=1157797
HDFS: Number of bytes written=884212
HDFS: Number of read operations=8576
HDFS: Number of large read operations=0
HDFS: Number of write operations=4224
Map-Reduce Framework
Map input records=793
Map output records=793
Input split bytes=6848
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=42
Total committed heap usage (bytes)=12124160000
File Input Format Counters
Bytes Read=28599
File Output Format Counters
Bytes Written=21848
What I have done so far:
I have found a similar question here: Hadoop WordCount example stuck at map 100% reduce 0% and followed some of the advice given. In particular:
At one point I configured Yarn so that I could go into localhost:8088 and monitor the job. All the mappers worked correctly - no failures, and the job ended abruptly right after the last mapper was successful, that is no reducers were ever started. Mapper was shown as 100% and reduce as 0%.
Using this command:
cat /path/to/logs/*.log | grep ERROR
returned nothing.
As I can see the output of the mapper stage I believe the problem does not lie there.
I have tried debugging: putting print statements in the configure method of the Reduce and the reduce method itself. None of them got printed when I rerun the file.
Additional note: Since the algorithm I am using was published and is supposed to work, I believe the problem could possibly arise from the fact that the code is 3 years old and was written for Hadoop 0.20.2 version but I guess I should not be too sure of this.
I understand that this is a specific question but I hope that someone could point me in the right direction. I would be happy to include anything else you might find useful. Any help is greatly appreciated!
I built an distributed environment with a virtual box. It contains 3 nodes, 1 master node and 2 slave nodes.
I've already put some data on HDFS, and I executed the following command:
./hadoop/bin/hadoop jar hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount input/ output/
Here is the log I got after few hours:
15/05/03 20:48:30 INFO client.RMProxy: Connecting to ResourceManager at hadoop01/192.168.10.11:8032
15/05/03 20:48:31 INFO input.FileInputFormat: Total input paths to process : 1
15/05/03 20:48:31 INFO mapreduce.JobSubmitter: number of splits:1
15/05/03 20:48:31 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
15/05/03 20:48:31 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
15/05/03 20:48:31 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
15/05/03 20:48:31 INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
15/05/03 20:48:31 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
15/05/03 20:48:31 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
15/05/03 20:48:31 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
15/05/03 20:48:31 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
15/05/03 20:48:31 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
15/05/03 20:48:31 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
15/05/03 20:48:31 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
15/05/03 20:48:31 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
15/05/03 20:48:32 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1430656198359_0002
15/05/03 20:48:32 INFO impl.YarnClientImpl: Submitted application application_1430656198359_0002 to ResourceManager at hadoop01/192.168.10.11:8032
15/05/03 20:48:32 INFO mapreduce.Job: The url to track the job: http://hadoop01:8088/proxy/application_1430656198359_0002/
15/05/03 20:48:32 INFO mapreduce.Job: Running job: job_1430656198359_0002
Here's the code snippet that works fine:
Configuration conf = new Configuration();
//PROBLEM PART!!!!!
//conf.setBoolean("mapred.compress.map.output", true);
//conf.set("mapred.output.compression.type", "BLOCK");
//conf.setClass("mapred.map.output.compression.codec", GzipCodec.class, CompressionCodec.class);
Job job = new Job(conf, "WordCount");
job.setJarByClass(WordCount.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapperClass(WordCountMap.class);
job.setReducerClass(Reduce.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
But if I enable the PROBLEM PART in the code snippet above, the output of console will stuck at:
13/12/26 18:08:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/12/26 18:08:06 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/12/26 18:08:06 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
13/12/26 18:08:06 INFO input.FileInputFormat: Total input paths to process : 20
13/12/26 18:08:06 WARN snappy.LoadSnappy: Snappy native library not loaded
13/12/26 18:08:06 INFO mapred.JobClient: Running job: job_local1943436108_0001
13/12/26 18:08:06 INFO mapred.LocalJobRunner: Waiting for map tasks
13/12/26 18:08:06 INFO mapred.LocalJobRunner: Starting task: attempt_local1943436108_0001_m_000000_0
13/12/26 18:08:07 INFO util.ProcessTree: setsid exited with exit code 0
13/12/26 18:08:07 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin#731d2572
13/12/26 18:08:07 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/jude/input/capacity-scheduler.xml:0+7457
13/12/26 18:08:07 INFO mapred.MapTask: io.sort.mb = 100
13/12/26 18:08:07 INFO mapred.MapTask: data buffer = 79691776/99614720
13/12/26 18:08:07 INFO mapred.MapTask: record buffer = 262144/327680
13/12/26 18:08:07 INFO mapred.MapTask: Starting flush of map output
13/12/26 18:08:07 INFO compress.CodecPool: Got brand-new compressor
13/12/26 18:08:07 INFO mapred.MapTask: Starting flush of map output
13/12/26 18:08:07 INFO mapred.JobClient: map 0% reduce 0%
13/12/26 18:08:12 INFO mapred.LocalJobRunner:
13/12/26 18:08:13 INFO mapred.JobClient: map 5% reduce 0%
//no more
I just intend to compress the output of map, is there anything wrong with my code? Thanks a lot!
Using compression requires Hadoop to use native libraries for your platform, but apparently you don't have them (or have not configured the path to the libraries correctly). This is the message that is explaining the problem:
[...] NativeCodeLoader: Unable to load native-hadoop library for your platform...
Possible solutions:
Most common problem is to have the 32-bit libraries on a 64-bit architecture. You can download the pre-compiled 64 native libraries OR compile them yourself using mvn package -Pdist,**native**,docs.
Or, you may need to configure the path to the native libraries correctly; see these other questions on how to do this: use -Djava.libray.path, or LD_LIBRARY_PATH.