We will be discussing interaction with HDFS using the command-line interface. For this, we can use two command prefaces:
➡ hadoop fs
➡ hdfs dfs
There are some basic differences between hadoop fs and hdfs dfs, but for all practical purposes of this post, both are the same.
List files/directories
➡ hadoop fs -ls <dirname> (Directory name has to be specified in HDFS)
➡ hadoop fs -ls -l -t -r <dirname>
-l = shows descriptive list and all file information
-t = sort by time
-r = reverse the input list
➡ hadoop fs -ls -S <dirname> (applicable for files only)
➡ hadoop fs -ls -l -h <dirname> (lists all files with human readable sizes)
➡ hadoop fs -ls -R <dirname> (recursively list all files/directories under the specified directory)
Create and remove HDFS directories
Create new directory
➡ hadoop fs -mkdir -p <dirname> (write full directory name. There is no concept of relative naming)
-p create if not exists
Delete directory
➡ hadoop fs -rm <filename>
➡ hadoop fs -rmdir <dirname> (remove empty directory)
➡ hadoop fs -rm -R <dirname> (remove full directory. recursive removal)
Copy files from local to HDFS
➡ hadoop fs -copyFromLocal <filename> <dirname> (file in local system to HDFS directory)
OR
hadoop fs -put <file/dirname> <dirname> (file/directory in local system to HDFS directory)
Copy files from HDFS to local
➡ hadoop fs -copyToLocal <filename> <dirname> (file in HDFS to local directory)
OR
hadoop fs -get <file/dirname> <dirname> (file/directory in HDFS to local directory)
Inter-HDFS file copy and move
➡ hadoop fs -cp <src_file> <dest_dirname>
➡ hadoop fs -mv <src_file> <dest_file/dirname> (can be moved to a new directory or same directory(for renaming))