org.apache.ignite.internal.processors.hadoop.impl

Class HadoopUtils

java.lang.Object

org.apache.ignite.internal.processors.hadoop.impl.HadoopUtils

public class HadoopUtils
extends Object

Hadoop utility methods.

Method Summary

Modifier and Type	Method and Description
static int	compareBytes(byte[] buf1, int len1, long ptr2, int len2) Internal comparison routine.
static HadoopDefaultJobInfo	createJobInfo(Configuration cfg, byte[] credentials) Creates JobInfo from hadoop configuration.
static UserGroupInformation	createUGI(String user, byte[] credentialsBytes) Create UserGroupInformation for specified user and credentials.
static void	deserialize(Writable writable, byte[] bytes) Deserialization of Hadoop Writable object.
static void	ensureNotSet(Configuration cfg, String attr, String msg) Checks the attribute in configuration is not set.
static Path	jobFile(Configuration conf, String usr, JobID jobId) Gets job file.
static File	jobLocalDir(String workDir, UUID locNodeId, HadoopJobId jobId) Returns work directory for job execution.
static Configuration	safeCreateConfiguration() Creates Configuration in a correct class loader context to avoid caching of inappropriate class loader in the Configuration object.
static Path	stagingAreaDir(Configuration conf, String usr) Gets staging area directory.
static JobStatus	status(HadoopJobStatus status, Configuration conf) Convert Ignite job status to Hadoop job status.
static File	taskLocalDir(String workDir, UUID locNodeId, HadoopTaskInfo info) Returns subdirectory of job working directory for task execution.
static IgniteCheckedException	transformException(Throwable e) Throws new IgniteCheckedException with original exception is serialized into string.
static Object	unwrapSplit(HadoopSplitWrapper o) Unwraps native split.
static HadoopSplitWrapper	wrapSplit(int id, Object split, String[] hosts) Wraps native split.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

wrapSplit

public static HadoopSplitWrapper wrapSplit(int id,
                                           Object split,
                                           String[] hosts)
                                    throws IOException

Wraps native split.

Parameters:

id - Split ID.

split - Split.

hosts - Hosts.

Throws:

IOException - If failed.

unwrapSplit

public static Object unwrapSplit(HadoopSplitWrapper o)

Unwraps native split.

Parameters:

o - Wrapper.

Returns:

Split.

status

public static JobStatus status(HadoopJobStatus status,
                               Configuration conf)

Convert Ignite job status to Hadoop job status.

Parameters:

status - Ignite job status.

Returns:

Hadoop job status.

stagingAreaDir

public static Path stagingAreaDir(Configuration conf,
                                  String usr)

Gets staging area directory.

Parameters:

conf - Configuration.

usr - User.

Returns:

Staging area directory.

jobFile

public static Path jobFile(Configuration conf,
                           String usr,
                           JobID jobId)

Gets job file.

Parameters:

conf - Configuration.

usr - User.

jobId - Job ID.

Returns:

Job file.

ensureNotSet

public static void ensureNotSet(Configuration cfg,
                                String attr,
                                String msg)
                         throws IgniteCheckedException

Checks the attribute in configuration is not set.

Parameters:

attr - Attribute name.

msg - Message for creation of exception.

Throws:

IgniteCheckedException - If attribute is set.

createJobInfo

public static HadoopDefaultJobInfo createJobInfo(Configuration cfg,
                                                 byte[] credentials)
                                          throws IgniteCheckedException

Creates JobInfo from hadoop configuration.

Parameters:

cfg - Hadoop configuration.

credentials - Credentials.

Returns:

Job info.

Throws:

IgniteCheckedException - If failed.

transformException

public static IgniteCheckedException transformException(Throwable e)

Throws new IgniteCheckedException with original exception is serialized into string. This is needed to transfer error outside the current class loader.

Parameters:

e - Original exception.

Returns:

IgniteCheckedException New exception.

jobLocalDir

public static File jobLocalDir(String workDir,
                               UUID locNodeId,
                               HadoopJobId jobId)
                        throws IgniteCheckedException

Returns work directory for job execution.

Parameters:

workDir - Work directory.

locNodeId - Local node ID.

jobId - Job ID.

Returns:

Working directory for job.

Throws:

IgniteCheckedException - If Failed.

taskLocalDir

public static File taskLocalDir(String workDir,
                                UUID locNodeId,
                                HadoopTaskInfo info)
                         throws IgniteCheckedException

Returns subdirectory of job working directory for task execution.

Parameters:

workDir - Work directory.

locNodeId - Local node ID.

info - Task info.

Returns:

Working directory for task.

Throws:

IgniteCheckedException - If Failed.

safeCreateConfiguration

public static Configuration safeCreateConfiguration()

Creates Configuration in a correct class loader context to avoid caching of inappropriate class loader in the Configuration object.

Returns:

New instance of Configuration.

compareBytes

public static int compareBytes(byte[] buf1,
                               int len1,
                               long ptr2,
                               int len2)

Internal comparison routine.

Parameters:

buf1 - Bytes 1.

len1 - Length 1.

ptr2 - Pointer 2.

len2 - Length 2.

Returns:

Result.

deserialize

public static void deserialize(Writable writable,
                               byte[] bytes)
                        throws IOException

Deserialization of Hadoop Writable object.

Parameters:

writable - Writable object to deserialize to.

bytes - byte array to deserialize.

Throws:

IOException

createUGI

public static UserGroupInformation createUGI(String user,
                                             byte[] credentialsBytes)
                                      throws IOException

Create UserGroupInformation for specified user and credentials.

Parameters:

user - User.

credentialsBytes - Credentials byte array.

Throws:

IOException