Group pig jobs by Hadoop step

[demikl]

On a Hadoop cluster used for several Pig computations, all jobs are displayed in the last Run Pig Script section, instead of having each job attached to its related pig script.

Consider this example :

┆   Steps
┆   ┆   Setup Hadoop Debugging
┆   ┆   ┆   Status:                COMPLETED
┆   ┆   ┆   Init/start/end date:   2013-08-07 16:15:47  /  2013-08-07 16:20:43 (+00:04:56)  /  2013-08-07 16:20:55 (+00:00:12)
┆   ┆   Setup Pig
┆   ┆   ┆   Status:                COMPLETED
┆   ┆   ┆   Init/start/end date:   2013-08-07 16:15:47  /  2013-08-07 16:20:55 (+00:05:08)  /  2013-08-07 16:21:41 (+00:00:46)
┆   ┆   Run Pig Script
┆   ┆   ┆   Status:                COMPLETED
┆   ┆   ┆   Init/start/end date:   2013-08-07 16:15:47  /  2013-08-07 16:21:00 (+00:05:13)  /  2013-08-07 16:21:05 (+00:00:05)
┆   ┆   ┆   Script:                s3://appnexus-us/input/src/videology.pig
┆   ┆   ┆   Input/output (size):    (–)  ⇒   (–)
┆   ┆   ┆   Other parameters:      –
┆   ┆   Run Pig Script
┆   ┆   ┆   Status:                RUNNING
┆   ┆   ┆   Init/start/end date:   2013-08-07 16:15:47  /  2013-08-07 16:21:41 (+00:05:54)  /  – (≈ +04:00:57)
┆   ┆   ┆   Script:                s3://xxx/yyy.pig
┆   ┆   ┆   Input/output (size):   s3://xxx/zzz1 (875.4 GiB)  ⇒  s3://xxx/zzz2 (–)
┆   ┆   ┆   Other parameters:      PARALLEL=240
┆   ┆   ┆   Cluster summary:       running maps/capacity: 768/768, running reduces/capacity: 0/256
┆   ┆   ┆   Jobs
┆   ┆   ┆   ┆   job_201308071418_0001   start date: 2013-08-07 16:22:31, maps completed: 1990/1990 (100.00%), reduces completed: 1/1 (100.00%)
┆   ┆   ┆   ┆   job_201308071418_0002   start date: 2013-08-07 17:31:43, maps completed: 3121/3121 (100.00%), reduces completed: 240/240 (100.00%)
┆   ┆   ┆   ┆   job_201308071418_0003   start date: 2013-08-07 20:21:02, maps completed: 8/5095 (1.63%), reduces completed: 0/240 (0.00%)

Let's consider job_201308071418_0001 is related to the first pig step. A better monitoring view would be:

┆   Steps
┆   ┆   Setup Hadoop Debugging
┆   ┆   ┆   Status:                COMPLETED
┆   ┆   ┆   Init/start/end date:   2013-08-07 16:15:47  /  2013-08-07 16:20:43 (+00:04:56)  /  2013-08-07 16:20:55 (+00:00:12)
┆   ┆   Setup Pig
┆   ┆   ┆   Status:                COMPLETED
┆   ┆   ┆   Init/start/end date:   2013-08-07 16:15:47  /  2013-08-07 16:20:55 (+00:05:08)  /  2013-08-07 16:21:41 (+00:00:46)
┆   ┆   Run Pig Script
┆   ┆   ┆   Status:                COMPLETED
┆   ┆   ┆   Init/start/end date:   2013-08-07 16:15:47  /  2013-08-07 16:21:00 (+00:05:13)  /  2013-08-07 16:21:05 (+00:00:05)
┆   ┆   ┆   Script:                s3://appnexus-us/input/src/videology.pig
┆   ┆   ┆   Input/output (size):    (–)  ⇒   (–)
┆   ┆   ┆   Other parameters:      –
┆   ┆   ┆   Jobs
┆   ┆   ┆   ┆   job_201308071418_0001   start date: 2013-08-07 16:22:31, maps completed: 1990/1990 (100.00%), reduces completed: 1/1 (100.00%)
┆   ┆   Run Pig Script
┆   ┆   ┆   Status:                RUNNING
┆   ┆   ┆   Init/start/end date:   2013-08-07 16:15:47  /  2013-08-07 16:21:41 (+00:05:54)  /  – (≈ +04:00:57)
┆   ┆   ┆   Script:                s3://xxx/yyy.pig
┆   ┆   ┆   Input/output (size):   s3://xxx/zzz1 (875.4 GiB)  ⇒  s3://xxx/zzz2 (–)
┆   ┆   ┆   Other parameters:      PARALLEL=240
┆   ┆   ┆   Cluster summary:       running maps/capacity: 768/768, running reduces/capacity: 0/256
┆   ┆   ┆   Jobs
┆   ┆   ┆   ┆   job_201308071418_0002   start date: 2013-08-07 17:31:43, maps completed: 3121/3121 (100.00%), reduces completed: 240/240 (100.00%)
┆   ┆   ┆   ┆   job_201308071418_0003   start date: 2013-08-07 20:21:02, maps completed: 8/5095 (1.63%), reduces completed: 0/240 (0.00%)

Amazon debug view in the EMR console is able to attach each job to its parent step. I've not yet determined where this link could be deduced in the various sources of information.