Combine HDFS and Linux Command to Query Data in Big Data Application

Introduction HDFS command and Linux command for text processing

Here We are presenting a tutorial about the combination of Hadoop Distributed File System Command (HDFS) And Linux Command using querying data in Big Data. Read the full post and you can try out it yourself.

HDFS command includes command which interacts with HDFS and another file system in Hadoop. For instance, we can use some basic commands to move, copy and delete the data folder in HDFS. The main purpose of this blog guide how to query, scan the data in HDFS with the HDFS command.

In Linux command, we use some basic commands to interact with text data such as grep, sort, uniq, etc. It is helpful with the local file but how we can use these commands with HDFS.

This blog will guide how to combine the advantages of commands to query the data in HDFS.

Environment

Java: JDK 1.7

Cloudera version: CDH4.6.0

Initial steps

1. We need to ready some input data file for user information about music, open a new file in the terminal of Linux:

vifile1

Text some input data with format: id;music;viewTime;duration;price;musicType

2. We need to put the local files to Hadoop Distributed File System (HDFS), use this command:

To run automation for the query above, we can add it all into one sh file and run it from the command file or add the sh file to a cron job to make everything in automation mode.

vi auto.sh

Hope that you guys can understand how to combine the Linux shell command and HDFS command to query the data in HDFS without Hive mapping. We use this approach in case we need some ad-hoc query.

All the instances and coding shared by Big Data Solutions experts from India & Hadoop architect for reference purposes only. You can ask your questions (if any) to them and get answers sooner.