HDFS Basic Commands
- Listing files and directories in HDFS – hadoop fs -ls /user/hdpuser (this is an HDFS directory)
-t = sort by time
-r = list in reverse
-S = sort by size
-R = recursively show all files under that particular directory(in this case, /user/hdpuser)
- Making new directories in HDFS – hadoop fs -mkdir /user/hdpuser/dir1
-p = create if not exists
- Removing files and directories in HDFS – hadoop fs -rm /user/hdpuser/dir1/newfile.txt
hadoop fs -rmdir /user/hdpuser/dir1 (remove empty directory)
hadoop fs -rm -R /user/hdpuser/dir2 (non empty directory)
- Copy files in HDFS – hadoop fs -cp <source_file> <dest_directory>
- Move/Rename files in HDFS – Same command used to both move and rename files
hadoop fs -mv <source_file> <dest_directory> (move file to different directory)
hadoop fs -mv <source_file> <new_name_same_file> (renaming)
File transfer to and from HDFS
Local System to HDFS
Using copyFromLocal – hadoop fs -copyFromLocal /home/localuser/file1.txt /user/hdpuser/data (first is local file path and second is HDFS destination)
Using put – hadoop fs -put /home/localuser/file1.txt /user/hdpuser/data
HDFS to Local System
Using copyToLocal – hadoop fs -copyToLocal /user/hdpuser/file1.csv /user/localuser/
Using get – hadoop fs -get /user/hdpuser/file1.csv /user/localuser/