| 论文简介 |
This paper firstly analyzes and points out the reasons of smallfile problem of HDFS: (1) large numbers of smallfiles impose heavy burden on NameNode of HDFS; (2) correlations between smallfiles are not considered for data placement; and (3) no optimization mechanism, such as prefetching, is provided to improve I/O performance. Secondly, in the context of HDFS, the clear cut-off point between large and smallfiles is determined through experimentation, which helps determine ‘how small is small’. Thirdly, according to file correlation features, files are classified into three types: structurally-related files, logically-related files, and independent files. Finally, based on the above three steps, an optimizedapproach is designed to improve the storage and access efficiencies of smallfiles on HDFS. File merging and prefetching scheme is applied for structurally-related smallfiles, while file grouping and prefetching scheme is used for managing logically-related smallfiles. Experimental results demonstrate that the proposed schemes effectively improve the storage and access efficiencies of smallfiles, compared with native HDFS and a Hadoop file archiving facility.（http://www.sciencedirect.com/science/article/pii/S1084804512001610） |
(创新港)
