Apache Oozie

Apache Oozie: Open Source Workflow Scheduling

Apache Oozie is an open source workflow scheduling and coordination system for managing Hadoop jobs. It allows users to define workflows that describe multi-stage Hadoop jobs and then execute those jobs in a dependable, repeatable fashion.

What is Apache Oozie?

Apache Oozie is an open source workflow scheduler system to manage Hadoop jobs. It is designed to run workflow jobs which represent a directed acyclic graph (DAG) of actions. Oozie workflows are written in hPDL (a XML Process Definition Language) and runs job instances based on the workflow definitions.

Key capabilities of Oozie include:

• Workflow scheduling and management of Hadoop jobs
• Support for different Hadoop jobs like Java map-reduce, Streaming map-reduce, Pig, Hive, Sqoop and Distcp
• DAG based workflow definition with fork and join semantics
• Retries of failed workflow actions
• Event notifications for workflow events
• Authentication, authorization and multi-tenancy

Oozie runs workflows following the defined DAG semantics based on the workflow application. The workflows can trigger Hadoop jobs when prerequisite jobs are completed. Oozie handles failures and retries of workflow actions. It also provides facilities to store workflow and action data for historical auditing.

Oozie is widely used for complex workflow scheduling in enterprise Hadoop deployments. It integrates well with Hadoop stack and provides a scalable solution to manage thousands of workflow jobs.

Apache Oozie Features

Official Links