An Innovative Strategy for Improved
Processing of Small Files in Hadoop

Priyanka Phakade, Dr. Suhas Raut

ABSTRACT Nowadays, the use of internet grows, so user wish to store data on cloud computing platform. Most of the time, userâ€™s data are small files. HDFS designed to process large volume of data. But it cannot handle large number of small files. In this paper, we have designed improved model for processing small files. In existing system, map tasks processes a block of input at a time. Map task produces intermediate output which is given to reducer. Reducer gives sort-merge output. Here, multiple map tasks & single reduce task is used for processing small file. In this approach, if there are large numbers of small files then each map task gets less input. In this way, performance of HDFS for processing lot of small files has been degraded. In proposed system, small files efficiently processed by HDFS. HDFS client requested to store small files in HDFS. NameNode permits to store small files in HDFS. When small files are submitted by HDFS client for processing, NameNode combine files into single split which becomes an input to map task. Intermediate output produced by map task is given to multiple reducers as an input. Reducer gives sorted merge output. As the number of map tasks has reduced, the processing time gets decreases. Keywords: Hadoop Distributed File System, Hadoop, MapReduce, Small File Problem

Title:	An Innovative Strategy for Improved Processing of Small Files in Hadoop
Author Name:	Priyanka Phakade, Dr. Suhas Raut
Abstract:	ABSTRACT Nowadays, the use of internet grows, so user wish to store data on cloud computing platform. Most of the time, userâ€™s data are small files. HDFS designed to process large volume of data. But it cannot handle large number of small files. In this paper, we have designed improved model for processing small files. In existing system, map tasks processes a block of input at a time. Map task produces intermediate output which is given to reducer. Reducer gives sort-merge output. Here, multiple map tasks & single reduce task is used for processing small file. In this approach, if there are large numbers of small files then each map task gets less input. In this way, performance of HDFS for processing lot of small files has been degraded. In proposed system, small files efficiently processed by HDFS. HDFS client requested to store small files in HDFS. NameNode permits to store small files in HDFS. When small files are submitted by HDFS client for processing, NameNode combine files into single split which becomes an input to map task. Intermediate output produced by map task is given to multiple reducers as an input. Reducer gives sorted merge output. As the number of map tasks has reduced, the processing time gets decreases. Keywords: Hadoop Distributed File System, Hadoop, MapReduce, Small File Problem
Cite this article:	Priyanka Phakade, Dr. Suhas Raut , " An Innovative Strategy for Improved Processing of Small Files in Hadoop " , International Journal of Application or Innovation in Engineering & Management (IJAIEM), Volume 3, Issue 7, July 2014 , pp. 278-280 , ISSN 2319 - 4847.
Full Text [PDF] Home