|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectorg.apache.hadoop.mapred.TaskTracker
public class TaskTracker
TaskTracker is a process that starts and tracks MR Tasks in a networked environment. It contacts the JobTracker for Task assignments and reporting results.
Nested Class Summary | |
---|---|
static class |
TaskTracker.MapOutputServlet
This class is used in TaskTracker's Jetty to serve the map outputs to other nodes. |
Field Summary | |
---|---|
static org.apache.commons.logging.Log |
ClientTraceLog
|
static int |
CLUSTER_INCREMENT
|
static long |
COUNTER_UPDATE_INTERVAL
|
static int |
FILE_NOT_FOUND
|
static java.lang.String |
FOR_REDUCE_TASK
The reduce task number for which this map output is being transferred |
static java.lang.String |
FROM_MAP_TASK
The map task from which the map output data is being transferred |
static int |
HEARTBEAT_INTERVAL_MIN
|
static org.apache.commons.logging.Log |
LOG
|
static java.lang.String |
MAP_OUTPUT_LENGTH
The custom http header used for the map output length. |
static java.lang.String |
MR_CLIENTTRACE_FORMAT
|
static java.lang.String |
RAW_MAP_OUTPUT_LENGTH
The custom http header used for the "raw" map output length. |
static int |
SUCCESS
|
static long |
versionID
Changed the version to 2, since we have a new method getMapOutputs Changed version to 3 to have progress() return a boolean Changed the version to 4, since we have replaced TaskUmbilicalProtocol.progress(String, float, String, org.apache.hadoop.mapred.TaskStatus.Phase, Counters) with statusUpdate(String, TaskStatus) Version 5 changed counters representation for HADOOP-2248 Version 6 changes the TaskStatus representation for HADOOP-2208 Version 7 changes the done api (via HADOOP-3140). |
static java.lang.String |
WORKDIR
|
Constructor Summary | |
---|---|
TaskTracker(JobConf conf)
Start with the local machine name, and the default JobTracker |
Method Summary | |
---|---|
boolean |
canCommit(TaskAttemptID taskid)
Child checking whether it can commit |
void |
cleanupStorage()
Removes all contents of temporary storage. |
void |
close()
Close down the TaskTracker and all its components. |
void |
commitPending(TaskAttemptID taskid,
org.apache.hadoop.mapred.TaskStatus taskStatus)
Task is reporting that it is in commit_pending and it is waiting for the commit Response |
void |
done(TaskAttemptID taskid)
The task is done. |
void |
fatalError(TaskAttemptID taskId,
java.lang.String msg)
A child task had a fatal error. |
void |
fsError(TaskAttemptID taskId,
java.lang.String message)
A child task had a local filesystem error. |
static java.lang.Class<? extends org.apache.hadoop.mapred.TaskTrackerInstrumentation> |
getInstrumentationClass(Configuration conf)
|
org.apache.hadoop.mapred.InterTrackerProtocol |
getJobClient()
The connection to the JobTracker, used by the TaskRunner for locating remote files. |
org.apache.hadoop.mapred.JvmManager |
getJvmManagerInstance()
|
org.apache.hadoop.mapred.MapTaskCompletionEventsUpdate |
getMapCompletionEvents(JobID jobId,
int fromEventId,
int maxLocs,
TaskAttemptID id)
Called by a reduce task to get the map output locations for finished maps. |
long |
getProtocolVersion(java.lang.String protocol,
long clientVersion)
Return protocol version corresponding to protocol interface. |
org.apache.hadoop.mapred.JvmTask |
getTask(org.apache.hadoop.mapred.JVMId jvmId)
Called upon startup by the child process, to fetch Task data. |
org.apache.hadoop.mapred.TaskMemoryManagerThread |
getTaskMemoryManager()
|
org.apache.hadoop.mapred.TaskTrackerInstrumentation |
getTaskTrackerInstrumentation()
|
java.net.InetSocketAddress |
getTaskTrackerReportAddress()
Return the port at which the tasktracker bound to |
boolean |
isIdle()
Is this task tracker idle? |
boolean |
isTaskMemoryManagerEnabled()
Is the TaskMemoryManager Enabled on this system? |
static void |
main(java.lang.String[] argv)
Start the TaskTracker, point toward the indicated JobTracker |
void |
mapOutputLost(TaskAttemptID taskid,
java.lang.String errorMsg)
A completed map task's output has been lost. |
boolean |
ping(TaskAttemptID taskid)
Child checking to see if we're alive. |
void |
reportDiagnosticInfo(TaskAttemptID taskid,
java.lang.String info)
Called when the task dies before completion, and we want to report back diagnostic info |
void |
reportNextRecordRange(TaskAttemptID taskid,
org.apache.hadoop.mapred.SortedRanges.Range range)
Report the record range which is going to process next by the Task. |
void |
run()
The server retry loop. |
static void |
setInstrumentationClass(Configuration conf,
java.lang.Class<? extends org.apache.hadoop.mapred.TaskTrackerInstrumentation> t)
|
void |
shuffleError(TaskAttemptID taskId,
java.lang.String message)
A reduce-task failed to shuffle the map-outputs. |
void |
shutdown()
|
boolean |
statusUpdate(TaskAttemptID taskid,
org.apache.hadoop.mapred.TaskStatus taskStatus)
Called periodically to report Task progress, from 0.0 to 1.0. |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
public static final org.apache.commons.logging.Log LOG
public static final java.lang.String MR_CLIENTTRACE_FORMAT
public static final org.apache.commons.logging.Log ClientTraceLog
public static final int HEARTBEAT_INTERVAL_MIN
public static final int CLUSTER_INCREMENT
public static final long COUNTER_UPDATE_INTERVAL
public static final int SUCCESS
public static final int FILE_NOT_FOUND
public static final java.lang.String MAP_OUTPUT_LENGTH
public static final java.lang.String RAW_MAP_OUTPUT_LENGTH
public static final java.lang.String FROM_MAP_TASK
public static final java.lang.String FOR_REDUCE_TASK
public static final java.lang.String WORKDIR
public static final long versionID
Constructor Detail |
---|
public TaskTracker(JobConf conf) throws java.io.IOException
java.io.IOException
Method Detail |
---|
public org.apache.hadoop.mapred.TaskTrackerInstrumentation getTaskTrackerInstrumentation()
public long getProtocolVersion(java.lang.String protocol, long clientVersion) throws java.io.IOException
VersionedProtocol
protocol
- The classname of the protocol interfaceclientVersion
- The version of the protocol that the client speaks
java.io.IOException
public static java.lang.Class<? extends org.apache.hadoop.mapred.TaskTrackerInstrumentation> getInstrumentationClass(Configuration conf)
public static void setInstrumentationClass(Configuration conf, java.lang.Class<? extends org.apache.hadoop.mapred.TaskTrackerInstrumentation> t)
public void cleanupStorage() throws java.io.IOException
java.io.IOException
public void shutdown() throws java.io.IOException
java.io.IOException
public void close() throws java.io.IOException
java.io.IOException
public org.apache.hadoop.mapred.InterTrackerProtocol getJobClient()
public java.net.InetSocketAddress getTaskTrackerReportAddress()
public org.apache.hadoop.mapred.JvmManager getJvmManagerInstance()
public void run()
run
in interface java.lang.Runnable
public org.apache.hadoop.mapred.JvmTask getTask(org.apache.hadoop.mapred.JVMId jvmId) throws java.io.IOException
jvmId
- the ID of this JVM w.r.t the tasktracker that launched it
java.io.IOException
public boolean statusUpdate(TaskAttemptID taskid, org.apache.hadoop.mapred.TaskStatus taskStatus) throws java.io.IOException
taskid
- task-id of the childtaskStatus
- status of the child
java.io.IOException
public void reportDiagnosticInfo(TaskAttemptID taskid, java.lang.String info) throws java.io.IOException
taskid
- the id of the task involvedinfo
- the text to report
java.io.IOException
public void reportNextRecordRange(TaskAttemptID taskid, org.apache.hadoop.mapred.SortedRanges.Range range) throws java.io.IOException
taskid
- the id of the task involvedrange
- the range of record sequence nos
java.io.IOException
public boolean ping(TaskAttemptID taskid) throws java.io.IOException
java.io.IOException
public void commitPending(TaskAttemptID taskid, org.apache.hadoop.mapred.TaskStatus taskStatus) throws java.io.IOException
taskid
- task's idtaskStatus
- status of the child
java.io.IOException
public boolean canCommit(TaskAttemptID taskid)
public void done(TaskAttemptID taskid) throws java.io.IOException
taskid
- task's id
java.io.IOException
public void shuffleError(TaskAttemptID taskId, java.lang.String message) throws java.io.IOException
java.io.IOException
public void fsError(TaskAttemptID taskId, java.lang.String message) throws java.io.IOException
java.io.IOException
public void fatalError(TaskAttemptID taskId, java.lang.String msg) throws java.io.IOException
java.io.IOException
public org.apache.hadoop.mapred.MapTaskCompletionEventsUpdate getMapCompletionEvents(JobID jobId, int fromEventId, int maxLocs, TaskAttemptID id) throws java.io.IOException
fromEventId
- the index starting from which the locations should be
fetchedmaxLocs
- the max number of locations to fetchid
- The attempt id of the task that is trying to communicate
MapTaskCompletionEventsUpdate
java.io.IOException
public void mapOutputLost(TaskAttemptID taskid, java.lang.String errorMsg) throws java.io.IOException
java.io.IOException
public boolean isIdle()
public static void main(java.lang.String[] argv) throws java.lang.Exception
java.lang.Exception
public boolean isTaskMemoryManagerEnabled()
public org.apache.hadoop.mapred.TaskMemoryManagerThread getTaskMemoryManager()
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |