Sqoop

Sqoop: Bulk Data Transfer

Open source tool for transferring data between Apache Hadoop and structured datastores

What is Sqoop?

Sqoop is an open source tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases. It provides a command-line interface that allows importing data from relational databases such as MySQL, Oracle, PostgreSQL into the Hadoop Distributed File System (HDFS), as well as exporting data from HDFS into relational databases.

Some key capabilities of Sqoop include:

• Full load: Sqoop can load entire tables/datasets from an RDBMS to HDFS.
• Incremental load: Sqoop can load only rows that were added/updated after latest import.
• Parallel data transfer: Sqoop imports data in parallel for optimized performance.
• Compression: Sqoop supports compression for minimizing network loads.
• Connectors: Sqoop provides connectivity to many popular RDBMS databases like MySQL, PostgreSQL, Oracle, SQL Server.
• Easy to use: Simple, straight-forward command-line interface.
• Portability: Sqoop works across a range of Hadoop versions and RDBMS.

By allowing bulk data transfers between Hadoop and relational databases, Sqoop enables real-time and batch-oriented processing of the same data under one platform. It is widely used by enterprises to move big data between Hadoop and production systems.

Sqoop Features

Official Links