We Keep Coding

Can anyone explain my Apache Spark Error SparkException: Job aborted due to stage failure

I have simple Apache Spark App where I read files from hdfs and after that i pipe it to external process. When I read a big amount a data (in my case files have about 241MB) and i don't specify min number of partitions or specify min number to 4 i'm getting following error:
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 6, ip-172-31-36-43.us-west-2.compute.internal): ExecutorLostFailure (executor 6 lost)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1191)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
When I specify min number of partitions to 10 or above i don't getting this error. Can anyone tell me what's wrong and avoid it? I didn't get error that subprocess exited with error code so I think it's problem with Spark configuration.
stderr from worker:
15/05/03 10:41:29 INFO executor.CoarseGrainedExecutorBackend: Registered signal handlers for [TERM, HUP, INT]
15/05/03 10:41:30 INFO spark.SecurityManager: Changing view acls to: root
15/05/03 10:41:30 INFO spark.SecurityManager: Changing modify acls to: root
15/05/03 10:41:30 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
15/05/03 10:41:30 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/05/03 10:41:30 INFO Remoting: Starting remoting
15/05/03 10:41:31 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://driverPropsFetcher#ip-172-31-36-43.us-west-2.compute.internal:46832]
15/05/03 10:41:31 INFO util.Utils: Successfully started service 'driverPropsFetcher' on port 46832.
15/05/03 10:41:31 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
15/05/03 10:41:31 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
15/05/03 10:41:31 INFO spark.SecurityManager: Changing view acls to: root
15/05/03 10:41:31 INFO spark.SecurityManager: Changing modify acls to: root
15/05/03 10:41:31 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
15/05/03 10:41:31 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
15/05/03 10:41:31 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/05/03 10:41:31 INFO Remoting: Starting remoting
15/05/03 10:41:31 INFO util.Utils: Successfully started service 'sparkExecutor' on port 37039.
15/05/03 10:41:31 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkExecutor#ip-172-31-36-43.us-west-2.compute.internal:37039]
15/05/03 10:41:31 INFO util.AkkaUtils: Connecting to MapOutputTracker: akka.tcp://sparkDriver#ip-172-31-35-111.us-west-2.compute.internal:48730/user/MapOutputTracker
15/05/03 10:41:31 INFO util.AkkaUtils: Connecting to BlockManagerMaster: akka.tcp://sparkDriver#ip-172-31-35-111.us-west-2.compute.internal:48730/user/BlockManagerMaster
15/05/03 10:41:31 INFO storage.DiskBlockManager: Created local directory at /mnt/spark/spark-cbaf9bff-4d12-4847-9135-9667ba27dccb/spark-ad82597c-4b55-46fc-9063-5d1196d6e0b0/spark-e99f55c6-5bcb-4d1b-b014-aaec94fe6cc5/blockmgr-cda1922d-ea50-4630-a834-bfb637ecdaa0
15/05/03 10:41:31 INFO storage.DiskBlockManager: Created local directory at /mnt2/spark/spark-0c6c912f-3aa1-4c54-9970-7a75d22899e8/spark-71d64ae7-36bc-49e0-958e-e7e2c1432027/spark-56d9e077-4585-4fd7-8a48-5227943d9004/blockmgr-29c5d068-f19d-4f41-85fc-11960c77a8a3
15/05/03 10:41:31 INFO storage.MemoryStore: MemoryStore started with capacity 445.4 MB
15/05/03 10:41:32 INFO util.AkkaUtils: Connecting to OutputCommitCoordinator: akka.tcp://sparkDriver#ip-172-31-35-111.us-west-2.compute.internal:48730/user/OutputCommitCoordinator
15/05/03 10:41:32 INFO executor.CoarseGrainedExecutorBackend: Connecting to driver: akka.tcp://sparkDriver#ip-172-31-35-111.us-west-2.compute.internal:48730/user/CoarseGrainedScheduler
15/05/03 10:41:32 INFO worker.WorkerWatcher: Connecting to worker akka.tcp://sparkWorker#ip-172-31-36-43.us-west-2.compute.internal:54983/user/Worker
15/05/03 10:41:32 INFO worker.WorkerWatcher: Successfully connected to akka.tcp://sparkWorker#ip-172-31-36-43.us-west-2.compute.internal:54983/user/Worker
15/05/03 10:41:32 INFO executor.CoarseGrainedExecutorBackend: Successfully registered with driver
15/05/03 10:41:32 INFO executor.Executor: Starting executor ID 6 on host ip-172-31-36-43.us-west-2.compute.internal
15/05/03 10:41:32 INFO netty.NettyBlockTransferService: Server created on 33000
15/05/03 10:41:32 INFO storage.BlockManagerMaster: Trying to register BlockManager
15/05/03 10:41:32 INFO storage.BlockManagerMaster: Registered BlockManager
15/05/03 10:41:32 INFO util.AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver#ip-172-31-35-111.us-west-2.compute.internal:48730/user/HeartbeatReceiver
15/05/03 10:41:32 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 6
15/05/03 10:41:32 INFO executor.Executor: Running task 1.3 in stage 0.0 (TID 6)
15/05/03 10:41:32 INFO executor.Executor: Fetching http://172.31.35.111:34347/jars/proteinsApacheSpark-0.0.1.jar with timestamp 1430649374764
15/05/03 10:41:32 INFO util.Utils: Fetching http://172.31.35.111:34347/jars/proteinsApacheSpark-0.0.1.jar to /mnt/spark/spark-cbaf9bff-4d12-4847-9135-9667ba27dccb/spark-ad82597c-4b55-46fc-9063-5d1196d6e0b0/spark-08b3b4ce-960f-488f-99ea-bd66b3277207/fetchFileTemp3079113313084659984.tmp
15/05/03 10:41:32 INFO util.Utils: Copying /mnt/spark/spark-cbaf9bff-4d12-4847-9135-9667ba27dccb/spark-ad82597c-4b55-46fc-9063-5d1196d6e0b0/spark-08b3b4ce-960f-488f-99ea-bd66b3277207/9655652641430649374764_cache to /root/spark/work/app-20150503103615-0002/6/./proteinsApacheSpark-0.0.1.jar
15/05/03 10:41:32 INFO executor.Executor: Adding file:/root/spark/work/app-20150503103615-0002/6/./proteinsApacheSpark-0.0.1.jar to class loader
15/05/03 10:41:32 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 1
15/05/03 10:41:32 INFO storage.MemoryStore: ensureFreeSpace(17223) called with curMem=0, maxMem=467081625
15/05/03 10:41:32 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 16.8 KB, free 445.4 MB)
15/05/03 10:41:32 INFO storage.BlockManagerMaster: Updated info of block broadcast_1_piece0
15/05/03 10:41:32 INFO broadcast.TorrentBroadcast: Reading broadcast variable 1 took 274 ms
15/05/03 10:41:32 INFO storage.MemoryStore: ensureFreeSpace(22384) called with curMem=17223, maxMem=467081625
15/05/03 10:41:32 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 21.9 KB, free 445.4 MB)
15/05/03 10:41:33 INFO spark.CacheManager: Partition rdd_0_1 not found, computing it
15/05/03 10:41:33 INFO rdd.WholeTextFileRDD: Input split: Paths:/user/root/pepnovo3/largeinputfile2/largeinputfile2_45.mgf:0+2106005,/user/root/pepnovo3/largeinputfile2/largeinputfile2_46.mgf:0+2105954,/user/root/pepnovo3/largeinputfile2/largeinputfile2_47.mgf:0+2106590,/user/root/pepnovo3/largeinputfile2/largeinputfile2_48.mgf:0+2105696,/user/root/pepnovo3/largeinputfile2/largeinputfile2_49.mgf:0+2105891,/user/root/pepnovo3/largeinputfile2/largeinputfile2_5.mgf:0+2106283,/user/root/pepnovo3/largeinputfile2/largeinputfile2_50.mgf:0+2105559,/user/root/pepnovo3/largeinputfile2/largeinputfile2_51.mgf:0+2106403,/user/root/pepnovo3/largeinputfile2/largeinputfile2_52.mgf:0+2105535,/user/root/pepnovo3/largeinputfile2/largeinputfile2_53.mgf:0+2105615,/user/root/pepnovo3/largeinputfile2/largeinputfile2_54.mgf:0+2105861,/user/root/pepnovo3/largeinputfile2/largeinputfile2_55.mgf:0+2106100,/user/root/pepnovo3/largeinputfile2/largeinputfile2_56.mgf:0+2106265,/user/root/pepnovo3/largeinputfile2/largeinputfile2_57.mgf:0+2105768,/user/root/pepnovo3/largeinputfile2/largeinputfile2_58.mgf:0+2106180,/user/root/pepnovo3/largeinputfile2/largeinputfile2_59.mgf:0+2105751,/user/root/pepnovo3/largeinputfile2/largeinputfile2_6.mgf:0+2106247,/user/root/pepnovo3/largeinputfile2/largeinputfile2_60.mgf:0+2106133,/user/root/pepnovo3/largeinputfile2/largeinputfile2_61.mgf:0+2106224,/user/root/pepnovo3/largeinputfile2/largeinputfile2_62.mgf:0+2106415,/user/root/pepnovo3/largeinputfile2/largeinputfile2_63.mgf:0+2106408,/user/root/pepnovo3/largeinputfile2/largeinputfile2_64.mgf:0+2105702,/user/root/pepnovo3/largeinputfile2/largeinputfile2_65.mgf:0+2106268,/user/root/pepnovo3/largeinputfile2/largeinputfile2_66.mgf:0+2106149,/user/root/pepnovo3/largeinputfile2/largeinputfile2_67.mgf:0+2105846,/user/root/pepnovo3/largeinputfile2/largeinputfile2_68.mgf:0+2105408,/user/root/pepnovo3/largeinputfile2/largeinputfile2_69.mgf:0+2106172,/user/root/pepnovo3/largeinputfile2/largeinputfile2_7.mgf:0+2105517,/user/root/pepnovo3/largeinputfile2/largeinputfile2_70.mgf:0+2105980,/user/root/pepnovo3/largeinputfile2/largeinputfile2_71.mgf:0+2105651,/user/root/pepnovo3/largeinputfile2/largeinputfile2_72.mgf:0+2105936,/user/root/pepnovo3/largeinputfile2/largeinputfile2_73.mgf:0+2105966,/user/root/pepnovo3/largeinputfile2/largeinputfile2_74.mgf:0+2105456,/user/root/pepnovo3/largeinputfile2/largeinputfile2_75.mgf:0+2105786,/user/root/pepnovo3/largeinputfile2/largeinputfile2_76.mgf:0+2106151,/user/root/pepnovo3/largeinputfile2/largeinputfile2_77.mgf:0+2106284,/user/root/pepnovo3/largeinputfile2/largeinputfile2_78.mgf:0+2106163,/user/root/pepnovo3/largeinputfile2/largeinputfile2_79.mgf:0+2106233,/user/root/pepnovo3/largeinputfile2/largeinputfile2_8.mgf:0+2105885,/user/root/pepnovo3/largeinputfile2/largeinputfile2_80.mgf:0+2105979,/user/root/pepnovo3/largeinputfile2/largeinputfile2_81.mgf:0+2105888,/user/root/pepnovo3/largeinputfile2/largeinputfile2_82.mgf:0+2106546,/user/root/pepnovo3/largeinputfile2/largeinputfile2_83.mgf:0+2106322,/user/root/pepnovo3/largeinputfile2/largeinputfile2_84.mgf:0+2106017,/user/root/pepnovo3/largeinputfile2/largeinputfile2_85.mgf:0+2106242,/user/root/pepnovo3/largeinputfile2/largeinputfile2_86.mgf:0+2105543,/user/root/pepnovo3/largeinputfile2/largeinputfile2_87.mgf:0+2106556,/user/root/pepnovo3/largeinputfile2/largeinputfile2_88.mgf:0+2105637,/user/root/pepnovo3/largeinputfile2/largeinputfile2_89.mgf:0+2106130,/user/root/pepnovo3/largeinputfile2/largeinputfile2_9.mgf:0+2105634,/user/root/pepnovo3/largeinputfile2/largeinputfile2_90.mgf:0+2105731,/user/root/pepnovo3/largeinputfile2/largeinputfile2_91.mgf:0+2106401,/user/root/pepnovo3/largeinputfile2/largeinputfile2_92.mgf:0+2105736,/user/root/pepnovo3/largeinputfile2/largeinputfile2_93.mgf:0+2105688,/user/root/pepnovo3/largeinputfile2/largeinputfile2_94.mgf:0+2106436,/user/root/pepnovo3/largeinputfile2/largeinputfile2_95.mgf:0+2105609,/user/root/pepnovo3/largeinputfile2/largeinputfile2_96.mgf:0+2105525,/user/root/pepnovo3/largeinputfile2/largeinputfile2_97.mgf:0+2105603,/user/root/pepnovo3/largeinputfile2/largeinputfile2_98.mgf:0+2106211,/user/root/pepnovo3/largeinputfile2/largeinputfile2_99.mgf:0+2105928
15/05/03 10:41:33 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 0
15/05/03 10:41:33 INFO storage.MemoryStore: ensureFreeSpace(6906) called with curMem=39607, maxMem=467081625
15/05/03 10:41:33 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 6.7 KB, free 445.4 MB)
15/05/03 10:41:33 INFO storage.BlockManagerMaster: Updated info of block broadcast_0_piece0
15/05/03 10:41:33 INFO broadcast.TorrentBroadcast: Reading broadcast variable 0 took 15 ms
15/05/03 10:41:33 INFO storage.MemoryStore: ensureFreeSpace(53787) called with curMem=46513, maxMem=467081625
15/05/03 10:41:33 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 52.5 KB, free 445.3 MB)
15/05/03 10:41:33 WARN snappy.LoadSnappy: Snappy native library is available
15/05/03 10:41:33 INFO util.NativeCodeLoader: Loaded the native-hadoop library
15/05/03 10:41:33 INFO snappy.LoadSnappy: Snappy native library loaded
15/05/03 10:41:36 INFO storage.MemoryStore: ensureFreeSpace(252731448) called with curMem=100300, maxMem=467081625
15/05/03 10:41:36 INFO storage.MemoryStore: Block rdd_0_1 stored as values in memory (estimated size 241.0 MB, free 204.3 MB)
15/05/03 10:41:36 INFO storage.BlockManagerMaster: Updated info of block rdd_0_1

The answer is probably in the executor log, which is different from the worker log. Most likely it runs out of memory and either starts GC thrashing or dies from OOM. You could try running with more memory per executor if this is an option.

Check your system hard disk space, network and memory. spark write file in $SPARK_HOME/work. some time full hard disk space, no memory free or network issue.
if any exception you can see on your_machine:4040

hadoop java.io.IOException: while running namenode -format on OSX

I am getting following error while formatting namenode, I have tried using sudo su as mentioned in some other stack overflow solutions but i am still getting this error,please assist.
14/01/16 16:10:41 INFO util.GSet: Computing capacity for map INodeMap
14/01/16 16:10:41 INFO util.GSet: VM type = 64-bit
14/01/16 16:10:41 INFO util.GSet: 1.0% max memory = 889 MB
14/01/16 16:10:41 INFO util.GSet: capacity = 2^20 = 1048576 entries
14/01/16 16:10:41 INFO namenode.NameNode: Caching file names occuring more than 10 times
14/01/16 16:10:41 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
14/01/16 16:10:41 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
14/01/16 16:10:41 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
14/01/16 16:10:41 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
14/01/16 16:10:41 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
14/01/16 16:10:41 INFO util.GSet: Computing capacity for map Namenode Retry Cache
14/01/16 16:10:41 INFO util.GSet: VM type = 64-bit
14/01/16 16:10:41 INFO util.GSet: 0.029999999329447746% max memory = 889 MB
14/01/16 16:10:41 INFO util.GSet: capacity = 2^15 = 32768 entries
14/01/16 16:10:41 FATAL namenode.NameNode: Exception in namenode join
java.io.IOException: Cannot create directory /Users/hadoop/hadoop/bin/hdfs/namenode/current
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:301)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:523)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:544)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:147)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:837)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1213)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320)
14/01/16 16:10:41 INFO util.ExitUtil: Exiting with status 1
14/01/16 16:10:41 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode
I just tried creating directory /Users/hadoop/hadoop/bin/hdfs/namenode/currentbut i found out that bin directory has file named hdfsso its not allowing me to create a directory named hdfs in bin directory so not sure if I should over write this hdfs file or copy it to other location?

If the directory /Users/hadoop/hadoop/bin/hdfs/namenode/current doesn't exist, you can't create a file there, so make sure you create the directory first.
Ensure you that you have read and execute access to all child directories as well: chmod o+x /Users/hadoop/hadoop/bin/hdfs/namenode/current
Check if hadoop runs under a different username. To find this you can use this command from the command line: ps aux | grep hadoop

hadoop connection refused on port 9000

I want to setup a hadoop-cluster in pseudo-distributed mode for development. Trying to start the hadoop cluster fails due to refused connection on port 9000.
These are my configs (pretty standard):
site-core.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>~/hacking/hd-data/tmp</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>~/hacking/hd-data/snn</value>
</property>
</configuration>
hdfs-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>~/hacking/hd-data/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>~/hacking/hd-data/dn</value>
</property>
<property>
<name>dfs.permissions.supergroup</name>
<value>hadoop</value>
</property>
</configuration>
haddop-env.sh - here I changed the config to IPv4 mode only (see last line):
# Set Hadoop-specific environment variables here.
# The only required environment variable is JAVA_HOME. All others are
# optional. When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.
# The java implementation to use. Required.
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
# Extra Java CLASSPATH elements. Optional.
# export HADOOP_CLASSPATH=
# The maximum amount of heap to use, in MB. Default is 1000.
# export HADOOP_HEAPSIZE=2000
# Extra Java runtime options. Empty by default.
# export HADOOP_OPTS=-server
# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS"
# export HADOOP_TASKTRACKER_OPTS=
# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
# export HADOOP_CLIENT_OPTS
# Extra ssh options. Empty by default.
# export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HADOOP_CONF_DIR"
# Where log files are stored. $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
# File naming remote slave hosts. $HADOOP_HOME/conf/slaves by default.
# export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves
# host:path where hadoop code should be rsync'd from. Unset by default.
# export HADOOP_MASTER=master:/home/$USER/src/hadoop
# Seconds to sleep between slave commands. Unset by default. This
# can be useful in large clusters, where, e.g., slave rsyncs can
# otherwise arrive faster than the master can service them.
# export HADOOP_SLAVE_SLEEP=0.1
# The directory where pid files are stored. /tmp by default.
# export HADOOP_PID_DIR=/var/hadoop/pids
# A string representing this instance of hadoop. $USER by default.
# export HADOOP_IDENT_STRING=$USER
# The scheduling priority for daemon processes. See 'man nice'.
# export HADOOP_NICENESS=10
# Disabling IPv6 for HADOOP
export HADOOP_OPTS=-Djava.net.preferIPv4Stack=true
/etc/hosts:
127.0.0.1 localhost zaphod
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
But at the beginning after calling ./start-dfs.sh following lines are in the log files:
hadoop-pschmidt-datanode-zaphod.log
2013-08-19 21:21:59,430 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = zaphod/127.0.1.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.204.0
STARTUP_MSG: build = git://hrt8n35.cc1.ygridcore.net/ on branch branch-0.20-security-204 -r 65e258bf0813ac2b15bb4c954660eaf9e8fba141; compiled by 'hortonow' on Thu Aug 25 23:25:52 UTC 2011
************************************************************/
2013-08-19 21:22:03,950 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-08-19 21:22:04,052 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2013-08-19 21:22:04,064 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-08-19 21:22:04,065 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started
2013-08-19 21:22:07,054 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2013-08-19 21:22:07,060 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-08-19 21:22:08,709 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 0 time(s).
2013-08-19 21:22:09,710 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 1 time(s).
2013-08-19 21:22:10,711 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 2 time(s).
2013-08-19 21:22:11,712 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 3 time(s).
2013-08-19 21:22:12,712 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 4 time(s).
2013-08-19 21:22:13,713 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 5 time(s).
2013-08-19 21:22:14,714 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 6 time(s).
2013-08-19 21:22:15,714 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 7 time(s).
2013-08-19 21:22:16,715 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 8 time(s).
2013-08-19 21:22:17,716 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:9000. Already tried 9 time(s).
2013-08-19 21:22:17,717 INFO org.apache.hadoop.ipc.RPC: Server at localhost/127.0.0.1:9000 not available yet, Zzzzz...
hadoop-pschmidt-namenode-zaphod.log
2013-08-19 21:21:59,443 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = zaphod/127.0.1.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.204.0
STARTUP_MSG: build = git://hrt8n35.cc1.ygridcore.net/ on branch branch-0.20-security-204 -r 65e258bf0813ac2b15bb4c954660eaf9e8fba141; compiled by 'hortonow' on Thu Aug 25 23:25:52 UTC 2011
************************************************************/
2013-08-19 21:22:03,950 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-08-19 21:22:04,052 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2013-08-19 21:22:04,064 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-08-19 21:22:04,064 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started
2013-08-19 21:22:06,050 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2013-08-19 21:22:06,056 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-08-19 21:22:06,095 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
2013-08-19 21:22:06,097 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source NameNode registered.
2013-08-19 21:22:06,232 INFO org.apache.hadoop.hdfs.util.GSet: VM type = 64-bit
2013-08-19 21:22:06,234 INFO org.apache.hadoop.hdfs.util.GSet: 2% max memory = 17.77875 MB
2013-08-19 21:22:06,235 INFO org.apache.hadoop.hdfs.util.GSet: capacity = 2^21 = 2097152 entries
2013-08-19 21:22:06,235 INFO org.apache.hadoop.hdfs.util.GSet: recommended=2097152, actual=2097152
2013-08-19 21:22:06,748 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=pschmidt
2013-08-19 21:22:06,748 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=hadoop
2013-08-19 21:22:06,748 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true
2013-08-19 21:22:06,754 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.block.invalidate.limit=100
2013-08-19 21:22:06,768 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
2013-08-19 21:22:07,262 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStateMBean and NameNodeMXBean
2013-08-19 21:22:07,322 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
2013-08-19 21:22:07,326 INFO org.apache.hadoop.hdfs.server.common.Storage: Storage directory /home/pschmidt/hacking/hadoop-0.20.204.0/~/hacking/hd-data/nn does not exist.
2013-08-19 21:22:07,329 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /home/pschmidt/hacking/hadoop-0.20.204.0/~/hacking/hd-data/nn is in an inconsistent state: storage directory does not exist or is not accessible.
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:291)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:97)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:379)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:353)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:254)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:434)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1153)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1162)
2013-08-19 21:22:07,331 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /home/pschmidt/hacking/hadoop-0.20.204.0/~/hacking/hd-data/nn is in an inconsistent state: storage directory does not exist or is not accessible.
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:291)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:97)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:379)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:353)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:254)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:434)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1153)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1162)
2013-08-19 21:22:07,332 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at zaphod/127.0.1.1
************************************************************/
After reformatting the hdfs following output is displayed:
13/08/19 21:50:21 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = zaphod/127.0.0.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.20.204.0
STARTUP_MSG: build = git://hrt8n35.cc1.ygridcore.net/ on branch branch-0.20-security-204 -r 65e258bf0813ac2b15bb4c954660eaf9e8fba141; compiled by 'hortonow' on Thu Aug 25 23:25:52 UTC 2011
************************************************************/
Re-format filesystem in ~/hacking/hd-data/nn ? (Y or N) Y
13/08/19 21:50:26 INFO util.GSet: VM type = 64-bit
13/08/19 21:50:26 INFO util.GSet: 2% max memory = 17.77875 MB
13/08/19 21:50:26 INFO util.GSet: capacity = 2^21 = 2097152 entries
13/08/19 21:50:26 INFO util.GSet: recommended=2097152, actual=2097152
13/08/19 21:50:27 INFO namenode.FSNamesystem: fsOwner=pschmidt
13/08/19 21:50:27 INFO namenode.FSNamesystem: supergroup=hadoop
13/08/19 21:50:27 INFO namenode.FSNamesystem: isPermissionEnabled=true
13/08/19 21:50:27 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
13/08/19 21:50:27 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
13/08/19 21:50:27 INFO namenode.NameNode: Caching file names occuring more than 10 times
13/08/19 21:50:27 INFO common.Storage: Image file of size 110 saved in 0 seconds.
13/08/19 21:50:28 INFO common.Storage: Storage directory ~/hacking/hd-data/nn has been successfully formatted.
13/08/19 21:50:28 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at zaphod/127.0.0.1
************************************************************/
Using netstat -lpten | grep java :
tcp 0 0 0.0.0.0:50301 0.0.0.0:* LISTEN 1000 50995 9875/java
tcp 0 0 0.0.0.0:35471 0.0.0.0:* LISTEN 1000 51775 9639/java
tcp6 0 0 :::2181 :::* LISTEN 1000 20841 2659/java
tcp6 0 0 :::36743 :::* LISTEN 1000 20524 2659/java
Using netstat -lpten | grep 9000 returns nothing, assuming that there is no application bound to this designated port after all.
What else can I look for to get my hdfs up and running. Don't hesitate to ask for further logs and config files.
Thanks in advance.

Use absolute path for this and make sure the hadoop user has permissions to access this directory:-
<property>
<name>dfs.data.dir</name>
<value>~/hacking/hd-data/dn</value>
</property>
also make sure you format this path like
# hadoop namenode -format

Modify the core-sit.xml from
hdfs://localhost:9000
to
hdfs:// YOUR REAL MASTER IP ADDRESS:9000
e.g.
hdfs://192.168.111.10:9000
works for me!

The short and sweet answer is that your node service is not running. simply do
$HADOOP_HOME/bin/hdfs

you have to do the following steps:
hdfs namenode -format
start-all.cmd

Try command.
telnet localhost 9000
If telnet command returns connection refused then
start hadoop by:
$HADOOP_HOME/sbin/start-dfs.sh
Then by running telnet again you can see the connection is estabilished

enabling a hadoop scheduler(Resource-aware Adaptive Scheduler)

I want to enable adaptivescheduler in hadoop 0.20.203.0. I have a jar file from this scheduler. (I am sure that this jar file works properly). I put jar file in HADOOP_HOME/lib and I set HADOOP_CLASSPATH in hadoop-env.sh. and I set the required property of scheduler in mapred-site.xml. when I run my cluster, all of jobtracker, datanode,.... start. But when I go to UI of scheduler(http://localhost:50030/scheduler), I face with error 404. The jar file of this scheduler contains the core of hadoop-0.20.203.0.
what should I do to solve this problem? please help me. my jobtracker log is:
2013-07-27 01:22:29,333 INFO org.apache.hadoop.mapred.JobTracker: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host = master/192.168.0.112
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.203.0
STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011
************************************************************/
2013-07-27 01:22:29,527 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-07-27 01:22:29,537 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2013-07-27 01:22:29,538 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-07-27 01:22:29,538 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics system started
2013-07-27 01:22:29,781 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2013-07-27 01:22:29,784 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-07-27 01:22:29,785 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2013-07-27 01:22:29,796 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
2013-07-27 01:22:29,796 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2013-07-27 01:22:29,796 INFO org.apache.hadoop.mapred.JobTracker: Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2013-07-27 01:22:29,797 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2013-07-27 01:22:29,827 INFO org.apache.hadoop.mapred.JobTracker: Starting jobtracker with owner as maedeh
2013-07-27 01:22:29,852 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort9001 registered.
2013-07-27 01:22:29,853 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort9001 registered.
2013-07-27 01:22:29,856 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
2013-07-27 01:22:35,276 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2013-07-27 01:22:35,404 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2013-07-27 01:22:35,668 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50030
2013-07-27 01:22:35,669 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50030 webServer.getConnectors()[0].getLocalPort() returned 50030
2013-07-27 01:22:35,669 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50030
2013-07-27 01:22:35,669 INFO org.mortbay.log: jetty-6.1.x
2013-07-27 01:22:36,225 INFO org.mortbay.log: Started SelectChannelConnector#0.0.0.0:50030
2013-07-27 01:22:36,233 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
2013-07-27 01:22:36,234 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source JobTrackerMetrics registered.
2013-07-27 01:22:36,234 INFO org.apache.hadoop.mapred.JobTracker: JobTracker up at: 9001
2013-07-27 01:22:36,234 INFO org.apache.hadoop.mapred.JobTracker: JobTracker webserver: 50030
2013-07-27 01:22:36,366 INFO org.apache.hadoop.mapred.JobTracker: Cleaning up the system directory
2013-07-27 01:22:36,468 INFO org.apache.hadoop.mapred.JobHistory: Creating DONE folder at file:/home/maedeh/hadoop-0.20.203.0/logs/history/done
2013-07-27 01:22:36,478 INFO org.apache.hadoop.mapred.JobTracker: History server being initialized in embedded mode
2013-07-27 01:22:36,481 INFO org.apache.hadoop.mapred.JobHistoryServer: Started job history server at: localhost:50030
2013-07-27 01:22:36,481 INFO org.apache.hadoop.mapred.JobTracker: Job History Server web address: localhost:50030
2013-07-27 01:22:36,484 INFO org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is inactive
2013-07-27 01:22:36,782 INFO org.apache.hadoop.mapred.AdaptiveScheduler: Successfully configured AdaptiveScheduler
2013-07-27 01:22:36,782 INFO org.apache.hadoop.mapred.JobTracker: Refreshing hosts information
2013-07-27 01:22:36,791 INFO org.apache.hadoop.util.HostsFileReader: Setting the includes file to
2013-07-27 01:22:36,791 INFO org.apache.hadoop.util.HostsFileReader: Setting the excludes file to
2013-07-27 01:22:36,791 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2013-07-27 01:22:36,791 INFO org.apache.hadoop.mapred.JobTracker: Decommissioning 0 nodes
2013-07-27 01:22:36,802 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2013-07-27 01:22:36,802 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9001: starting
2013-07-27 01:22:36,803 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 9001: starting
2013-07-27 01:22:36,804 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001: starting
2013-07-27 01:22:36,804 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 9001: starting
2013-07-27 01:22:36,804 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 9001: starting
2013-07-27 01:22:36,804 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9001: starting
2013-07-27 01:22:36,805 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9001: starting
2013-07-27 01:22:36,805 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 9001: starting
2013-07-27 01:22:36,805 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 9001: starting
2013-07-27 01:22:36,805 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 9001: starting
2013-07-27 01:22:36,806 INFO org.apache.hadoop.mapred.JobTracker: Starting RUNNING
2013-07-27 01:22:36,806 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 9001: starting
2013-07-27 01:22:46,806 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/slave1
2013-07-27 01:22:46,808 INFO org.apache.hadoop.mapred.JobTracker: Adding tracker tracker_slave1:localhost/127.0.0.1:58226 to host slave1
2013-07-27 01:22:47,856 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/slave2
2013-07-27 01:22:47,859 INFO org.apache.hadoop.mapred.JobTracker: Adding tracker tracker_slave2:localhost/127.0.0.1:55061 to host slave2
2013-07-27 01:26:28,522 INFO org.apache.hadoop.mapred.JobInProgress: job_201307270122_0001: nMaps=3 nReduces=1 max=-1
2013-07-27 01:26:28,525 INFO org.apache.hadoop.mapred.JobTracker: Job job_201307270122_0001 added successfully for user 'maedeh' to queue 'default'
2013-07-27 01:26:28,538 INFO org.apache.hadoop.mapred.AuditLogger: USER=maedeh IP=192.168.0.112 OPERATION=SUBMIT_JOB TARGET=job_201307270122_0001 RESULT=SUCCESS
2013-07-27 01:26:28,560 INFO org.apache.hadoop.mapred.JobTracker: Initializing job_201307270122_0001
2013-07-27 01:26:28,560 INFO org.apache.hadoop.mapred.JobInProgress: Initializing job_201307270122_0001
2013-07-27 01:26:29,359 INFO org.apache.hadoop.mapred.JobInProgress: jobToken generated and stored with users keys in /home/maedeh/tempdir/mapred/system/job_201307270122_0001/jobToken
2013-07-27 01:26:29,403 INFO org.apache.hadoop.mapred.JobInProgress: Input size for job job_201307270122_0001 = 3671523. Number of splits = 3
2013-07-27 01:26:29,404 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201307270122_0001_m_000000 has split on node:/default-rack/slave1
2013-07-27 01:26:29,404 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201307270122_0001_m_000000 has split on node:/default-rack/slave2
2013-07-27 01:26:29,404 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201307270122_0001_m_000001 has split on node:/default-rack/slave1
2013-07-27 01:26:29,405 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201307270122_0001_m_000001 has split on node:/default-rack/slave2
2013-07-27 01:26:29,405 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201307270122_0001_m_000002 has split on node:/default-rack/slave1
2013-07-27 01:26:29,405 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201307270122_0001_m_000002 has split on node:/default-rack/slave2
2013-07-27 01:26:29,405 INFO org.apache.hadoop.mapred.JobInProgress: job_201307270122_0001 LOCALITY_WAIT_FACTOR=1.0
2013-07-27 01:26:29,405 INFO org.apache.hadoop.mapred.JobInProgress: Job job_201307270122_0001 initialized successfully with 3 map tasks and 1 reduce tasks.
2013-07-27 01:26:29,708 INFO org.apache.hadoop.mapred.JobTracker: Adding task (JOB_SETUP) 'attempt_201307270122_0001_m_000004_0' to tip task_201307270122_0001_m_000004, for tracker 'tracker_slave1:localhost/127.0.0.1:58226'
2013-07-27 01:26:39,051 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201307270122_0001_m_000004_0' has completed task_201307270122_0001_m_000004 successfully.
2013-07-27 01:26:39,069 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201307270122_0001_m_000000_0' to tip task_201307270122_0001_m_000000, for tracker 'tracker_slave1:localhost/127.0.0.1:58226'
2013-07-27 01:26:39,073 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201307270122_0001_m_000000
2013-07-27 01:26:40,326 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201307270122_0001_m_000001_0' to tip task_201307270122_0001_m_000001, for tracker 'tracker_slave2:localhost/127.0.0.1:55061'
2013-07-27 01:26:40,345 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201307270122_0001_m_000001
2013-07-27 01:26:42,214 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201307270122_0001_m_000002_0' to tip task_201307270122_0001_m_000002, for tracker 'tracker_slave1:localhost/127.0.0.1:58226'
2013-07-27 01:26:42,214 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201307270122_0001_m_000002
2013-07-27 01:27:00,452 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201307270122_0001_m_000000_0' has completed task_201307270122_0001_m_000000 successfully.
2013-07-27 01:27:01,759 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201307270122_0001_m_000001_0' has completed task_201307270122_0001_m_000001 successfully.
2013-07-27 01:27:06,476 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201307270122_0001_m_000002_0' has completed task_201307270122_0001_m_000002 successfully.
2013-07-27 01:27:09,536 INFO org.apache.hadoop.mapred.JobTracker: Adding task (REDUCE) 'attempt_201307270122_0001_r_000000_0' to tip task_201307270122_0001_r_000000, for tracker 'tracker_slave1:localhost/127.0.0.1:58226'
2013-07-27 01:27:21,749 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201307270122_0001_r_000000_0' has completed task_201307270122_0001_r_000000 successfully.
2013-07-27 01:27:21,756 INFO org.apache.hadoop.mapred.JobTracker: Adding task (JOB_CLEANUP) 'attempt_201307270122_0001_m_000003_0' to tip task_201307270122_0001_m_000003, for tracker 'tracker_slave1:localhost/127.0.0.1:58226'
2013-07-27 01:27:27,774 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201307270122_0001_m_000003_0' has completed task_201307270122_0001_m_000003 successfully.
2013-07-27 01:27:27,775 INFO org.apache.hadoop.mapred.JobInProgress: Job job_201307270122_0001 has completed successfully.
2013-07-27 01:27:27,790 INFO org.apache.hadoop.mapred.JobInProgress$JobSummary: jobId=job_201307270122_0001,submitTime=1374913588466,launchTime=1374913589405,firstMapTaskLaunchTime=1374913599068,firstReduceTaskLaunchTime=1374913629494,firstJobSetupTaskLaunchTime=1374913589678,firstJobCleanupTaskLaunchTime=1374913641756,finishTime=1374913647775,numMaps=3,numSlotsPerMap=1,numReduces=1,numSlotsPerReduce=1,user=maedeh,queue=default,status=SUCCEEDED,mapSlotSeconds=67,reduceSlotsSeconds=10,clusterMapCapacity=4,clusterReduceCapacity=4
2013-07-27 01:27:28,328 INFO org.apache.hadoop.mapred.JobHistory: Creating DONE subfolder at file:/home/maedeh/hadoop-0.20.203.0/logs/history/done/version-1/master_1374913354885_/2013/07/27/000000
2013-07-27 01:27:28,330 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/home/maedeh/hadoop-0.20.203.0/logs/history/job_201307270122_0001_1374913588466_maedeh_word+count to file:/home/maedeh/hadoop-0.20.203.0/logs/history/done/version-1/master_1374913354885_/2013/07/27/000000
2013-07-27 01:27:28,336 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/home/maedeh/hadoop-0.20.203.0/logs/history/job_201307270122_0001_conf.xml to file:/home/maedeh/hadoop-0.20.203.0/logs/history/done/version-1/master_1374913354885_/2013/07/27/000000
2013-07-27 01:27:28,345 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201307270122_0001_m_000000_0'
2013-07-27 01:27:28,348 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201307270122_0001_m_000002_0'
2013-07-27 01:27:28,348 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201307270122_0001_m_000003_0'
2013-07-27 01:27:28,348 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201307270122_0001_m_000004_0'
2013-07-27 01:27:28,348 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201307270122_0001_r_000000_0'
2013-07-27 01:27:29,228 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201307270122_0001_m_000001_0'
wordcount example:
maedeh#master:~/hadoop-0.20.203.0$ bin/hadoop jar hadoop*examples*.jar wordcount /maedeh/gutenberg /maedeh/gutenberg-output
13/07/27 01:26:27 INFO input.FileInputFormat: Total input paths to process : 3
13/07/27 01:26:28 INFO mapred.JobClient: Running job: job_201307270122_0001
13/07/27 01:26:29 INFO mapred.JobClient: map 0% reduce 0%
13/07/27 01:27:01 INFO mapred.JobClient: map 33% reduce 0%
13/07/27 01:27:03 INFO mapred.JobClient: map 66% reduce 0%
13/07/27 01:27:07 INFO mapred.JobClient: map 100% reduce 0%
13/07/27 01:27:22 INFO mapred.JobClient: map 100% reduce 100%
13/07/27 01:27:28 INFO mapred.JobClient: Job complete: job_201307270122_0001
13/07/27 01:27:28 INFO mapred.JobClient: Counters: 25
13/07/27 01:27:28 INFO mapred.JobClient: Job Counters
13/07/27 01:27:28 INFO mapred.JobClient: Launched reduce tasks=1
13/07/27 01:27:28 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=67684
13/07/27 01:27:28 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
13/07/27 01:27:28 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
13/07/27 01:27:28 INFO mapred.JobClient: Launched map tasks=3
13/07/27 01:27:28 INFO mapred.JobClient: Data-local map tasks=3
13/07/27 01:27:28 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=10249
13/07/27 01:27:28 INFO mapred.JobClient: File Output Format Counters
13/07/27 01:27:28 INFO mapred.JobClient: Bytes Written=880838
13/07/27 01:27:28 INFO mapred.JobClient: FileSystemCounters
13/07/27 01:27:28 INFO mapred.JobClient: FILE_BYTES_READ=2214875
13/07/27 01:27:28 INFO mapred.JobClient: HDFS_BYTES_READ=3671869
13/07/27 01:27:28 INFO mapred.JobClient: FILE_BYTES_WRITTEN=3775263
13/07/27 01:27:28 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=880838
13/07/27 01:27:28 INFO mapred.JobClient: File Input Format Counters
13/07/27 01:27:28 INFO mapred.JobClient: Bytes Read=3671523
13/07/27 01:27:28 INFO mapred.JobClient: Map-Reduce Framework
13/07/27 01:27:28 INFO mapred.JobClient: Reduce input groups=82335
13/07/27 01:27:28 INFO mapred.JobClient: Map output materialized bytes=1474367
13/07/27 01:27:28 INFO mapred.JobClient: Combine output records=102324
13/07/27 01:27:28 INFO mapred.JobClient: Map input records=77931
13/07/27 01:27:28 INFO mapred.JobClient: Reduce shuffle bytes=1474367
13/07/27 01:27:28 INFO mapred.JobClient: Reduce output records=82335
13/07/27 01:27:28 INFO mapred.JobClient: Spilled Records=255966
13/07/27 01:27:28 INFO mapred.JobClient: Map output bytes=6076101
13/07/27 01:27:28 INFO mapred.JobClient: Combine input records=629172
13/07/27 01:27:28 INFO mapred.JobClient: Map output records=629172
13/07/27 01:27:28 INFO mapred.JobClient: SPLIT_RAW_BYTES=346
13/07/27 01:27:28 INFO mapred.JobClient: Reduce input records=102324
maedeh#master:~/hadoop-0.20.203.0$

You can check which scheduler is running by going into jobtracker page->job history-> job file right side link->click on it->where you can get scheduler info which is running.

Output of variables of the hadoop program with the help of log4j

I wrote hadoop the program with log4j use (only Map step which operation didn't meet my waitings)
package org.myorg;
import java.io.*;
import java.util.*;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;
import org.apache.log4j.Logger;
import org.apache.log4j.LogManager;
import org.apache.log4j.xml.DOMConfigurator;
public class ParallelIndexation {
public static class Map extends MapReduceBase implements
Mapper<LongWritable, Text, Text, LongWritable> {
private final static LongWritable zero = new LongWritable(0);
private Text word = new Text();
private static final Logger logger = LogManager.getLogger(Map.class.getName());
public void map(LongWritable key, Text value,
OutputCollector<Text, LongWritable> output, Reporter reporter)
throws IOException {
DOMConfigurator.configure("/folder/log4j.xml");
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
Path localPath = new Path("/export/hadoop-1.0.1/bin/input/paths.txt");
Path hdfsPath=new Path("hdfs://192.168.1.8:7000/user/hadoop/paths.txt");
Path localPath1 = new Path("/usr/countcomputers.txt");
Path hdfsPath1=new Path("hdfs://192.168.1.8:7000/user/hadoop/countcomputers.txt");
if (!fs.exists(hdfsPath))
{
fs.copyFromLocalFile(localPath, hdfsPath);
};
if (!fs.exists(hdfsPath1))
{
fs.copyFromLocalFile(localPath1, hdfsPath1);
};
FSDataInputStream in = fs.open(hdfsPath);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
String line = br.readLine();
BufferedReader br1=new BufferedReader(new InputStreamReader(fs.open(hdfsPath1)));
int CountComputers;
String result=br1.readLine();
CountComputers=Integer.parseInt(result);
ArrayList<String> paths = new ArrayList<String>();
StringTokenizer tokenizer = new StringTokenizer(line, "|");
while (tokenizer.hasMoreTokens()) {
paths.add(tokenizer.nextToken());
}
for (int i=0; i<paths.size(); i++)
{
logger.debug("paths[i]=" + paths.get(i) + "\n");
}
logger.debug("CountComputers=" + CountComputers + "\n");
I provide the /folder/log4j.xml file
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration debug="true" xmlns:log4j="http://jakarta.apache.org/log4j/">
<appender name="ConsoleAppender" class="org.apache.log4j.ConsoleAppender">
<param name="Encoding" value="UTF-8"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{ISO8601} [%-5p][%-16.16t][%32.32c] - %m%n" />
</layout>
</appender>
<root>
<priority value="DEBUG"/>
<appender-ref ref="ConsoleAppender" />
</root>
</log4j:configuration>
But despite an output in a log, as a result of command execution
./hadoop jar /export/hadoop-1.0.1/bin/ParallelIndexation.jar org.myorg.ParallelIndexation /export/hadoop-1.0.1/bin/input /export/hadoop-1.0.1/bin/output -D mapred.map.tasks=1 1> resultofexecute.txt 2>&1
in the resultofexecute.txt file there was no output of appropriate variables.Help to remove the necessary variables.
#ChrisWhite I provide the hadoop-hadoop-tasktracker-myhost2.log and hadoop-hadoop-takstacker-myhost3.log files
hadoop-hadoop-tasktracker-myhost2.log
2013-04-20 12:35:43,465 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-04-20 12:35:43,583 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2013-04-20 12:35:43,584 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-04-20 12:35:43,584 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics system started
2013-04-20 12:35:43,971 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2013-04-20 12:35:43,979 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-04-20 12:35:44,168 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2013-04-20 12:35:44,251 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2013-04-20 12:35:44,280 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-04-20 12:35:44,287 INFO org.apache.hadoop.mapred.TaskTracker: Starting tasktracker with owner as hadoop
2013-04-20 12:35:44,288 INFO org.apache.hadoop.mapred.TaskTracker: Good mapred local directories are: /tmp/hadoop-hadoop/mapred/local
2013-04-20 12:35:44,302 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2013-04-20 12:35:44,312 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
2013-04-20 12:35:44,312 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source TaskTrackerMetrics registered.
2013-04-20 12:35:49,332 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort51172 registered.
2013-04-20 12:35:49,332 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort51172 registered.
2013-04-20 12:35:49,335 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost/127.0.0.1:51172
2013-04-20 12:35:49,335 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_myhost2:localhost/127.0.0.1:51172
2013-04-20 12:35:49,356 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
2013-04-20 12:35:49,357 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2013-04-20 12:35:49,357 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 51172: starting
2013-04-20 12:35:49,358 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 51172: starting
2013-04-20 12:35:49,358 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 51172: starting
2013-04-20 12:35:49,358 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 51172: starting
2013-04-20 12:35:49,358 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 51172: starting
2013-04-20 12:35:50,372 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 0 time(s).
2013-04-20 12:35:51,372 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 1 time(s).
2013-04-20 12:35:52,375 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 2 time(s).
2013-04-20 12:35:53,376 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 3 time(s).
2013-04-20 12:35:54,377 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 4 time(s).
2013-04-20 12:35:55,377 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 5 time(s).
2013-04-20 12:35:56,379 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 6 time(s).
2013-04-20 12:35:57,380 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 7 time(s).
2013-04-20 12:35:58,381 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 8 time(s).
2013-04-20 12:35:59,381 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 9 time(s).
2013-04-20 12:35:59,385 INFO org.apache.hadoop.ipc.RPC: Server at 192.168.1.8/192.168.1.8:7001 not available yet, Zzzzz...
2013-04-20 12:36:01,387 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 0 time(s).
2013-04-20 12:36:02,387 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 1 time(s).
2013-04-20 12:36:03,388 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 2 time(s).
2013-04-20 12:36:04,388 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 3 time(s).
2013-04-20 12:36:05,388 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 4 time(s).
2013-04-20 12:36:06,388 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 5 time(s).
2013-04-20 12:36:07,390 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 6 time(s).
2013-04-20 12:36:08,390 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 7 time(s).
2013-04-20 12:36:09,391 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 8 time(s).
2013-04-20 12:36:10,392 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 9 time(s).
2013-04-20 12:36:10,393 INFO org.apache.hadoop.ipc.RPC: Server at 192.168.1.8/192.168.1.8:7001 not available yet, Zzzzz...
2013-04-20 12:36:12,393 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 0 time(s).
2013-04-20 12:36:43,011 INFO org.apache.hadoop.mapred.TaskTracker: Using ResourceCalculatorPlugin : null
2013-04-20 12:36:43,030 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for all reduce tasks on tracker_myhost2:localhost/127.0.0.1:51172
2013-04-20 12:36:43,031 WARN org.apache.hadoop.util.ProcessTree: setsid is not available on this machine. So not using it.
2013-04-20 12:36:43,031 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
2013-04-20 12:36:43,182 INFO org.apache.hadoop.util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
2013-04-20 12:36:43,182 INFO org.apache.hadoop.mapred.TaskTracker: ProcessTree implementation is missing on this system. TaskMemoryManager is disabled.
2013-04-20 12:36:43,206 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760
2013-04-20 12:36:43,214 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ShuffleServerMetrics registered.
2013-04-20 12:36:43,217 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50060
2013-04-20 12:36:43,218 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50060 webServer.getConnectors()[0].getLocalPort() returned 50060
2013-04-20 12:36:43,218 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50060
2013-04-20 12:36:43,218 INFO org.mortbay.log: jetty-6.1.26
2013-04-20 12:36:43,611 INFO org.mortbay.log: Started SelectChannelConnector#0.0.0.0:50060
2013-04-20 12:36:43,611 INFO org.apache.hadoop.mapred.TaskTracker: FILE_CACHE_SIZE for mapOutputServlet set to : 2000
2013-04-20 12:36:43,635 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201304112319_0001 for user-log deletion with retainTimeStamp:1366573003186
2013-04-20 12:36:43,635 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201304070459_0003 for user-log deletion with retainTimeStamp:1366573003186
2013-04-20 12:36:43,635 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201304070459_0001 for user-log deletion with retainTimeStamp:1366573003186
2013-04-20 12:36:43,635 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201304112319_0007 for user-log deletion with retainTimeStamp:1366573003186
2013-04-20 12:36:43,635 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201304070413_0001 for user-log deletion with retainTimeStamp:1366573003186
2013-04-20 12:36:43,635 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201304121018_0001 for user-log deletion with retainTimeStamp:1366573003186
2013-04-20 12:36:43,635 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201304192207_0003 for user-log deletion with retainTimeStamp:1366573003186
2013-04-20 12:49:24,662 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201304201135_0001_m_000001_0 task's state:UNASSIGNED
2013-04-20 12:49:24,665 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201304201135_0001_m_000001_0 which needs 1 slots
2013-04-20 12:49:24,665 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201304201135_0001_m_000001_0 which needs 1 slots
2013-04-20 12:49:25,006 INFO org.apache.hadoop.mapred.JobLocalizer: Initializing user hadoop on this TT.
2013-04-20 12:49:25,377 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201304201135_0001_m_-1899194781
2013-04-20 12:49:25,378 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201304201135_0001_m_-1899194781 spawned.
2013-04-20 12:49:25,381 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-hadoop/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201304201135_0001/attempt_201304201135_0001_m_000001_0/taskjvm.sh
2013-04-20 12:49:26,284 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201304201135_0001_m_-1899194781 given task: attempt_201304201135_0001_m_000001_0
2013-04-20 12:49:29,803 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201304201135_0001_m_000001_0 0.0% setup
2013-04-20 12:49:29,806 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201304201135_0001_m_000001_0 is done.
2013-04-20 12:49:29,807 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201304201135_0001_m_000001_0 was -1
2013-04-20 12:49:29,809 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2013-04-20 12:49:29,947 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201304201135_0001_m_-1899194781 exited with exit code 0. Number of tasks it ran: 1
2013-04-20 12:49:30,719 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201304201135_0001_r_000000_0 task's state:UNASSIGNED
2013-04-20 12:49:30,719 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201304201135_0001_r_000000_0 which needs 1 slots
2013-04-20 12:49:30,719 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201304201135_0001_r_000000_0 which needs 1 slots
2013-04-20 12:49:30,719 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201304201135_0001_m_000001_0
2013-04-20 12:49:30,719 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201304201135_0001_m_000001_0
2013-04-20 12:49:30,720 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201304201135_0001_m_000001_0 not found in cache
2013-04-20 12:49:30,742 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201304201135_0001_r_-1899194781
2013-04-20 12:49:30,742 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201304201135_0001_r_-1899194781 spawned.
2013-04-20 12:49:30,745 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-hadoop/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201304201135_0001/attempt_201304201135_0001_r_000000_0/taskjvm.sh
2013-04-20 12:49:31,611 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201304201135_0001_r_-1899194781 given task: attempt_201304201135_0001_r_000000_0
2013-04-20 12:49:33,219 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201304201135_0001_r_000000_0 0.0%
2013-04-20 12:49:33,226 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201304201135_0001_r_000000_0 0.0%
2013-04-20 12:49:33,296 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201304201135_0001_r_000000_0 is in commit-pending, task state:COMMIT_PENDING
2013-04-20 12:49:33,297 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201304201135_0001_r_000000_0 0.0%
2013-04-20 12:49:33,731 INFO org.apache.hadoop.mapred.TaskTracker: Received commit task action for attempt_201304201135_0001_r_000000_0
2013-04-20 12:49:35,143 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201304201135_0001_r_000000_0 1.0% reduce > reduce
2013-04-20 12:49:35,147 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201304201135_0001_r_000000_0 is done.
2013-04-20 12:49:35,147 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201304201135_0001_r_000000_0 was -1
2013-04-20 12:49:35,155 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2013-04-20 12:49:35,272 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201304201135_0001_r_-1899194781 exited with exit code 0. Number of tasks it ran: 1
2013-04-20 12:49:36,775 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201304201135_0001_m_000000_0 task's state:UNASSIGNED
2013-04-20 12:49:36,783 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201304201135_0001_m_000000_0 which needs 1 slots
2013-04-20 12:49:36,783 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201304201135_0001_m_000000_0 which needs 1 slots
2013-04-20 12:49:36,783 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201304201135_0001_r_000000_0
2013-04-20 12:49:36,784 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201304201135_0001_r_000000_0
2013-04-20 12:49:36,812 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201304201135_0001_m_-653215379
2013-04-20 12:49:36,812 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201304201135_0001_m_-653215379 spawned.
2013-04-20 12:49:36,814 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-hadoop/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201304201135_0001/attempt_201304201135_0001_m_000000_0/taskjvm.sh
2013-04-20 12:49:37,718 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201304201135_0001_m_-653215379 given task: attempt_201304201135_0001_m_000000_0
2013-04-20 12:49:38,227 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201304201135_0001_m_000000_0 0.0%
2013-04-20 12:49:41,232 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201304201135_0001_m_000000_0 0.0% cleanup
2013-04-20 12:49:41,238 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201304201135_0001_m_000000_0 0.0% cleanup
2013-04-20 12:49:41,243 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201304201135_0001_m_000000_0 is done.
2013-04-20 12:49:41,243 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201304201135_0001_m_000000_0 was -1
2013-04-20 12:49:41,245 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2013-04-20 12:49:41,378 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201304201135_0001_m_-653215379 exited with exit code 0. Number of tasks it ran: 1
2013-04-20 12:49:42,821 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_201304201135_0001
2013-04-20 12:49:42,822 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201304201135_0001_m_000000_0 not found in cache
2013-04-20 12:49:42,826 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201304201135_0001 for user-log deletion with retainTimeStamp:1366573782822
hadoop-hadoop-tasktracker-myhost3.log
2013-04-20 12:35:43,798 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics system started
2013-04-20 12:35:44,200 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2013-04-20 12:35:44,207 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-04-20 12:35:44,382 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2013-04-20 12:35:44,467 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2013-04-20 12:35:44,496 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-04-20 12:35:44,500 INFO org.apache.hadoop.mapred.TaskTracker: Starting tasktracker with owner as hadoop
2013-04-20 12:35:44,501 INFO org.apache.hadoop.mapred.TaskTracker: Good mapred local directories are: /tmp/hadoop-hadoop/mapred/local
2013-04-20 12:35:44,506 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2013-04-20 12:35:44,520 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
2013-04-20 12:35:44,521 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source TaskTrackerMetrics registered.
2013-04-20 12:35:49,567 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort39327 registered.
2013-04-20 12:35:49,568 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort39327 registered.
2013-04-20 12:35:49,575 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost/127.0.0.1:39327
2013-04-20 12:35:49,575 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_myhost3:localhost/127.0.0.1:39327
2013-04-20 12:35:49,586 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
2013-04-20 12:35:49,587 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2013-04-20 12:35:49,587 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 39327: starting
2013-04-20 12:35:49,587 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 39327: starting
2013-04-20 12:35:49,587 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 39327: starting
2013-04-20 12:35:49,587 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 39327: starting
2013-04-20 12:35:49,587 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 39327: starting
2013-04-20 12:35:50,617 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 0 time(s).
2013-04-20 12:35:51,617 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 1 time(s).
2013-04-20 12:35:52,618 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 2 time(s).
2013-04-20 12:35:53,618 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 3 time(s).
2013-04-20 12:35:54,619 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 4 time(s).
2013-04-20 12:35:55,619 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 5 time(s).
2013-04-20 12:35:56,620 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 6 time(s).
2013-04-20 12:35:57,620 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 7 time(s).
2013-04-20 12:35:58,621 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 8 time(s).
2013-04-20 12:35:59,622 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 9 time(s).
2013-04-20 12:35:59,625 INFO org.apache.hadoop.ipc.RPC: Server at 192.168.1.8/192.168.1.8:7001 not available yet, Zzzzz...
2013-04-20 12:36:01,624 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 0 time(s).
2013-04-20 12:36:02,625 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 1 time(s).
2013-04-20 12:36:03,624 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 2 time(s).
2013-04-20 12:36:04,625 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 3 time(s).
2013-04-20 12:36:05,624 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 4 time(s).
2013-04-20 12:36:06,625 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 5 time(s).
2013-04-20 12:36:07,627 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 6 time(s).
2013-04-20 12:36:08,627 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 7 time(s).
2013-04-20 12:36:09,627 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 8 time(s).
2013-04-20 12:36:10,628 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 9 time(s).
2013-04-20 12:36:10,629 INFO org.apache.hadoop.ipc.RPC: Server at 192.168.1.8/192.168.1.8:7001 not available yet, Zzzzz...
2013-04-20 12:36:12,629 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 192.168.1.8/192.168.1.8:7001. Already tried 0 time(s).
2013-04-20 12:36:42,982 INFO org.apache.hadoop.mapred.TaskTracker: Using ResourceCalculatorPlugin : null
2013-04-20 12:36:43,000 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for all reduce tasks on tracker_myhost3:localhost/127.0.0.1:39327
2013-04-20 12:36:43,002 WARN org.apache.hadoop.util.ProcessTree: setsid is not available on this machine. So not using it.
2013-04-20 12:36:43,002 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
2013-04-20 12:36:43,157 INFO org.apache.hadoop.util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
2013-04-20 12:36:43,157 INFO org.apache.hadoop.mapred.TaskTracker: ProcessTree implementation is missing on this system. TaskMemoryManager is disabled.
2013-04-20 12:36:43,165 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760
2013-04-20 12:36:43,173 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ShuffleServerMetrics registered.
2013-04-20 12:36:43,176 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50060
2013-04-20 12:36:43,176 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50060 webServer.getConnectors()[0].getLocalPort() returned 50060
2013-04-20 12:36:43,176 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50060
2013-04-20 12:36:43,176 INFO org.mortbay.log: jetty-6.1.26
2013-04-20 12:36:43,567 INFO org.mortbay.log: Started SelectChannelConnector#0.0.0.0:50060
2013-04-20 12:36:43,567 INFO org.apache.hadoop.mapred.TaskTracker: FILE_CACHE_SIZE for mapOutputServlet set to : 2000
2013-04-20 12:36:43,576 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201304060623_0002 for user-log deletion with retainTimeStamp:1366573003161
2013-04-20 12:36:43,576 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201304060623_0005 for user-log deletion with retainTimeStamp:1366573003161
2013-04-20 12:36:43,576 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201304070459_0001 for user-log deletion with retainTimeStamp:1366573003161
2013-04-20 12:36:43,576 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201304030513_0001 for user-log deletion with retainTimeStamp:1366573003161
2013-04-20 12:36:43,576 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201304070413_0001 for user-log deletion with retainTimeStamp:1366573003161
2013-04-20 12:36:43,576 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201304012201_0015 for user-log deletion with retainTimeStamp:1366573003161
2013-04-20 12:36:43,576 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201304192207_0001 for user-log deletion with retainTimeStamp:1366573003161
2013-04-20 12:36:43,576 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201304121018_0003 for user-log deletion with retainTimeStamp:1366573003161
2013-04-20 12:36:43,576 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201304112319_0001 for user-log deletion with retainTimeStamp:1366573003161
2013-04-20 12:36:43,577 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201304112319_0005 for user-log deletion with retainTimeStamp:1366573003161
2013-04-20 12:36:43,577 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201304070459_0003 for user-log deletion with retainTimeStamp:1366573003161
2013-04-20 12:49:45,627 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_201304201135_0001
2013-04-20 12:49:45,628 WARN org.apache.hadoop.mapred.TaskTracker: Unknown job job_201304201135_0001 being deleted.

Log4j output will be written to the task log for each map / reduce task - this doesn't make its way back to the job client std out / err unless a map / reduce task fails in some way.
You'll need to find the task instance via the job tracker, and then view the logs for a particular map / reduce task instance.