org.apache.ignite.internal.processors.hadoop.shuffle

Class HadoopShuffleJob<T>

java.lang.Object

org.apache.ignite.internal.processors.hadoop.shuffle.HadoopShuffleJob<T>

All Implemented Interfaces:

AutoCloseable

public class HadoopShuffleJob<T>
extends Object
implements AutoCloseable

Shuffle job.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	HadoopShuffleJob.PartitionedOutput Partitioned output.

Field Summary

Fields

Modifier and Type	Field and Description
protected ConcurrentMap<Long,IgniteBiTuple<HadoopShuffleMessage,GridFutureAdapter<?>>>	sentMsgs

Constructor Summary

Constructors

Constructor and Description
HadoopShuffleJob(T locReduceAddr, IgniteLogger log, HadoopJobEx job, GridUnsafeMemory mem, int totalReducerCnt, int[] locReducers, int locMappersCnt, boolean embedded)

Method Summary

Modifier and Type	Method and Description
void	close()
IgniteInternalFuture<?>	flush()
boolean	initializeReduceAddresses(T[] reduceAddrs)
HadoopTaskInput	input(HadoopTaskContext taskCtx)
void	onDirectShuffleMessage(T src, HadoopDirectShuffleMessage msg) Process shuffle message.
void	onShuffleAck(HadoopShuffleAck ack)
void	onShuffleFinishRequest(T src, HadoopShuffleFinishRequest msg) Process shuffle finish request.
void	onShuffleFinishResponse(T src) Process shuffle finish response.
void	onShuffleMessage(T src, HadoopShuffleMessage msg)
HadoopTaskOutput	output(HadoopTaskContext taskCtx)
boolean	reducersInitialized()
void	startSending(String igniteInstanceName, IgniteInClosure2X<T,HadoopMessage> io)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

sentMsgs

protected ConcurrentMap<Long,IgniteBiTuple<HadoopShuffleMessage,GridFutureAdapter<?>>> sentMsgs

Constructor Detail

HadoopShuffleJob

public HadoopShuffleJob(T locReduceAddr,
                        IgniteLogger log,
                        HadoopJobEx job,
                        GridUnsafeMemory mem,
                        int totalReducerCnt,
                        int[] locReducers,
                        int locMappersCnt,
                        boolean embedded)
                 throws IgniteCheckedException

Parameters:

locReduceAddr - Local reducer address.

log - Logger.

job - Job.

mem - Memory.

totalReducerCnt - Amount of reducers in the Job.

locReducers - Reducers will work on current node.

locMappersCnt - Number of mappers running on the given node.

embedded - Whether shuffle is running in embedded mode.

Throws:

IgniteCheckedException - If error.

Method Detail

initializeReduceAddresses

public boolean initializeReduceAddresses(T[] reduceAddrs)

Parameters:

reduceAddrs - Addresses of reducers.

Returns:

True if addresses were initialized by this call.

reducersInitialized

public boolean reducersInitialized()

Returns:

True if reducers addresses were initialized.

startSending

public void startSending(String igniteInstanceName,
                         IgniteInClosure2X<T,HadoopMessage> io)

Parameters:

igniteInstanceName - Ignite instance name.

io - IO Closure for sending messages.

onShuffleMessage

public void onShuffleMessage(T src,
                             HadoopShuffleMessage msg)
                      throws IgniteCheckedException

Parameters:

src - Source.

msg - Message.

Throws:

IgniteCheckedException - Exception.

onDirectShuffleMessage

public void onDirectShuffleMessage(T src,
                                   HadoopDirectShuffleMessage msg)
                            throws IgniteCheckedException

Process shuffle message.

Parameters:

src - Source.

msg - Message.

Throws:

IgniteCheckedException - Exception.

onShuffleAck

public void onShuffleAck(HadoopShuffleAck ack)

Parameters:

ack - Shuffle ack.

onShuffleFinishRequest

public void onShuffleFinishRequest(T src,
                                   HadoopShuffleFinishRequest msg)

Process shuffle finish request.

Parameters:

src - Source.

msg - Shuffle finish message.

onShuffleFinishResponse

public void onShuffleFinishResponse(T src)

Process shuffle finish response.

Parameters:

src - Source.

close

public void close()
           throws IgniteCheckedException

Specified by:

close in interface AutoCloseable

Throws:

IgniteCheckedException

flush

public IgniteInternalFuture<?> flush()
                              throws IgniteCheckedException

Returns:

Future.

Throws:

IgniteCheckedException

output

public HadoopTaskOutput output(HadoopTaskContext taskCtx)
                        throws IgniteCheckedException

Parameters:

taskCtx - Task context.

Returns:

Output.

Throws:

IgniteCheckedException - If failed.

input

public HadoopTaskInput input(HadoopTaskContext taskCtx)
                      throws IgniteCheckedException

Parameters:

taskCtx - Task context.

Returns:

Input.

Throws:

IgniteCheckedException - If failed.