Skip to content

New Ionosphere Updates do not always complete in Anchorage (Track 131) #135

@cmarshak

Description

@cmarshak

Describe the bug
I submitted 15 frames over Anchorage AK. 5/15 failed during the iono step. Here is a sample log:

Number of threads:            4
 Complex data interpolation
 Initializing Sinc interpolator
 Azimuth Carrier Poly
Polynomial Order: 5 - by - 3 
11967.9	-575.794	31.7785	0.954085	
-47241.6	2301.56	-111.96	-3.57909	
46620	-2299.58	97.1936	3.37479	
-9.04525e-09	1.14979e-08	-4.21725e-09	0	
6.02973e-09	-3.23611e-09	0	0	
-1.51145e-09	0	0	0	
 Range Carrier Poly
Polynomial Order: 0 - by - 0 
0	
 Range offsets poly
Polynomial Order: 0 - by - 0 
0	
 Azimuth offsets poly
Polynomial Order: 0 - by - 0 
0	
 Doppler poly
Polynomial Order: 0 - by - 2 
0.0333558	-50.0337	82464.7	
 Reading in the image
 At line         1000
 Interpolating image
 At line         1000
 Elapsed time:    15.9726562    
GDAL close: fine_offsets/IW3/range_09.off.vrt
GDAL close: fine_offsets/IW3/azimuth_09.off.vrt
GDAL close: ion/upper/fine_interferogram/IW3/secondary_burst_09.slc.vrt
API close:  ion/upper/fine_interferogram/IW3/secondary_resamp_burst_09.slc
API open (R): ion/upper/fine_interferogram/IW3/secondary_resamp_burst_09.slc
API close:  ion/upper/fine_interferogram/IW3/secondary_resamp_burst_09.slc
Adjust valid samples
Before:  274 23521
Offsets :  11.545783996582031 12.257086753845215
Adjust valid lines
Before:  27 1468
Offsets :  -1.5180559158325195 -1.4518603086471558

  0%|          | 0/28 [00:00<?, ?it/s]
Geo2radar mask IW2 /10:   0%|          | 0/28 [00:00<?, ?it/s]
Geo2radar mask IW2 /10:   4%|â–Ž         | 1/28 [00:02<00:55,  2.04s/it]
Geo2radar mask IW2 /05:   4%|â–Ž         | 1/28 [00:02<00:55,  2.04s/it]
Geo2radar mask IW2 /05:   7%|â–‹         | 2/28 [00:04<01:03,  2.44s/it]
Geo2radar mask IW2 /01:   7%|â–‹         | 2/28 [00:04<01:03,  2.44s/it]
Geo2radar mask IW2 /01:  11%|â–ˆ         | 3/28 [00:07<00:59,  2.40s/it]
Geo2radar mask IW2 /03:  11%|â–ˆ         | 3/28 [00:07<00:59,  2.40s/it]
Geo2radar mask IW2 /03:  14%|█�        | 4/28 [00:09<00:57,  2.41s/it]
Geo2radar mask IW2 /08:  14%|█�        | 4/28 [00:09<00:57,  2.41s/it]
Geo2radar mask IW2 /08:  18%|█▊        | 5/28 [00:11<00:54,  2.38s/it]
Geo2radar mask IW2 /09:  18%|█▊        | 5/28 [00:11<00:54,  2.38s/it]
Geo2radar mask IW2 /09:  21%|██�       | 6/28 [00:14<00:52,  2.41s/it]
Geo2radar mask IW2 /06:  21%|██�       | 6/28 [00:14<00:52,  2.41s/it]
Geo2radar mask IW2 /06:  25%|██▌       | 7/28 [00:16<00:49,  2.37s/it]
Geo2radar mask IW2 /07:  25%|██▌       | 7/28 [00:16<00:49,  2.37s/it]
Geo2radar mask IW2 /07:  29%|██▊       | 8/28 [00:19<00:47,  2.39s/it]
Geo2radar mask IW2 /04:  29%|██▊       | 8/28 [00:19<00:47,  2.39s/it]
Geo2radar mask IW2 /04:  32%|███�      | 9/28 [00:21<00:45,  2.37s/it]
Geo2radar mask IW2 /02:  32%|███�      | 9/28 [00:21<00:45,  2.37s/it]
Geo2radar mask IW2 /02:  36%|███▌      | 10/28 [00:23<00:42,  2.38s/it]
Geo2radar mask IW1 /05:  36%|███▌      | 10/28 [00:23<00:42,  2.38s/it]
Geo2radar mask IW1 /05:  39%|███▉      | 11/28 [00:25<00:37,  2.18s/it]
Geo2radar mask IW1 /01:  39%|███▉      | 11/28 [00:25<00:37,  2.18s/it]
Geo2radar mask IW1 /01:  43%|████▎     | 12/28 [00:27<00:32,  2.04s/it]
Geo2radar mask IW1 /03:  43%|████▎     | 12/28 [00:27<00:32,  2.04s/it]
Geo2radar mask IW1 /03:  46%|████▋     | 13/28 [00:28<00:29,  1.95s/it]
Geo2radar mask IW1 /08:  46%|████▋     | 13/28 [00:28<00:29,  1.95s/it]
Geo2radar mask IW1 /08:  50%|█████     | 14/28 [00:30<00:26,  1.90s/it]
Geo2radar mask IW1 /09:  50%|█████     | 14/28 [00:30<00:26,  1.90s/it]
Geo2radar mask IW1 /09:  54%|█████▎    | 15/28 [00:32<00:24,  1.87s/it]
Geo2radar mask IW1 /06:  54%|█████▎    | 15/28 [00:32<00:24,  1.87s/it]
Geo2radar mask IW1 /06:  57%|█████▋    | 16/28 [00:34<00:22,  1.86s/it]
Geo2radar mask IW1 /07:  57%|█████▋    | 16/28 [00:34<00:22,  1.86s/it]
Geo2radar mask IW1 /07:  61%|██████    | 17/28 [00:36<00:20,  1.84s/it]
Geo2radar mask IW1 /04:  61%|██████    | 17/28 [00:36<00:20,  1.84s/it]
Geo2radar mask IW1 /04:  64%|██████�   | 18/28 [00:37<00:18,  1.81s/it]
Geo2radar mask IW1 /02:  64%|██████�   | 18/28 [00:37<00:18,  1.81s/it]
Geo2radar mask IW1 /02:  68%|██████▊   | 19/28 [00:39<00:16,  1.80s/it]
Geo2radar mask IW3 /05:  68%|██████▊   | 19/28 [00:39<00:16,  1.80s/it]
Geo2radar mask IW3 /05:  71%|███████�  | 20/28 [00:42<00:15,  1.98s/it]
Geo2radar mask IW3 /01:  71%|███████�  | 20/28 [00:42<00:15,  1.98s/it]
Geo2radar mask IW3 /01:  75%|███████▌  | 21/28 [00:44<00:14,  2.05s/it]
Geo2radar mask IW3 /03:  75%|███████▌  | 21/28 [00:44<00:14,  2.05s/it]
Geo2radar mask IW3 /03:  79%|███████▊  | 22/28 [00:46<00:12,  2.13s/it]
Geo2radar mask IW3 /08:  79%|███████▊  | 22/28 [00:46<00:12,  2.13s/it]
Geo2radar mask IW3 /08:  82%|████████� | 23/28 [00:48<00:10,  2.16s/it]
Geo2radar mask IW3 /09:  82%|████████� | 23/28 [00:48<00:10,  2.16s/it]
Geo2radar mask IW3 /09:  86%|████████▌ | 24/28 [00:51<00:08,  2.20s/it]
Geo2radar mask IW3 /06:  86%|████████▌ | 24/28 [00:51<00:08,  2.20s/it]
Geo2radar mask IW3 /06:  89%|████████▉ | 25/28 [00:53<00:06,  2.21s/it]
Geo2radar mask IW3 /07:  89%|████████▉ | 25/28 [00:53<00:06,  2.21s/it]
Geo2radar mask IW3 /07:  93%|█████████▎| 26/28 [00:55<00:04,  2.24s/it]
Geo2radar mask IW3 /04:  93%|█████████▎| 26/28 [00:55<00:04,  2.24s/it]
Geo2radar mask IW3 /04:  96%|█████████▋| 27/28 [00:57<00:02,  2.24s/it]
Geo2radar mask IW3 /02:  96%|█████████▋| 27/28 [00:57<00:02,  2.24s/it]
Geo2radar mask IW3 /02: 100%|██████████| 28/28 [01:00<00:00,  2.26s/it]
Geo2radar mask IW3 /02: 100%|██████████| 28/28 [01:00<00:00,  2.15s/it]

  0%|          | 0/56 [00:00<?, ?it/s]
Masking lower-iono interferograms IW2/10:   0%|          | 0/56 [00:00<?, ?it/s]
Masking lower-iono interferograms IW2/10:   2%|â–�         | 1/56 [00:00<00:30,  1.79it/s]
Masking lower-iono interferograms IW2/03:   2%|â–�         | 1/56 [00:00<00:30,  1.79it/s]
Masking lower-iono interferograms IW2/03:   4%|â–Ž         | 2/56 [00:01<00:42,  1.27it/s]
Masking lower-iono interferograms IW2/08:   4%|â–Ž         | 2/56 [00:01<00:42,  1.27it/s]
Masking lower-iono interferograms IW2/08:   5%|▌         | 3/56 [00:02<00:49,  1.06it/s]
Masking lower-iono interferograms IW2/02:   5%|▌         | 3/56 [00:02<00:49,  1.06it/s]
Masking lower-iono interferograms IW2/02:   7%|â–‹         | 4/56 [00:03<00:55,  1.07s/it]
Masking lower-iono interferograms IW2/04:   7%|â–‹         | 4/56 [00:03<00:55,  1.07s/it]
Masking lower-iono interferograms IW2/04:   9%|â–‰         | 5/56 [00:04<00:52,  1.03s/it]
Masking lower-iono interferograms IW2/05:   9%|â–‰         | 5/56 [00:04<00:52,  1.03s/it]
Masking lower-iono interferograms IW2/05:  11%|â–ˆ         | 6/56 [00:06<00:56,  1.13s/it]
Masking lower-iono interferograms IW2/06:  11%|â–ˆ         | 6/56 [00:06<00:56,  1.13s/it]
Masking lower-iono interferograms IW2/06:  12%|█▎        | 7/56 [00:07<01:05,  1.34s/it]
Masking lower-iono interferograms IW2/01:  12%|█▎        | 7/56 [00:07<01:05,  1.34s/it]
Masking lower-iono interferograms IW2/01:  14%|█�        | 8/56 [00:09<01:13,  1.53s/it]
Masking lower-iono interferograms IW2/07:  14%|█�        | 8/56 [00:09<01:13,  1.53s/it]
Masking lower-iono interferograms IW2/07:  16%|█▌        | 9/56 [00:11<01:18,  1.67s/it]
Masking lower-iono interferograms IW2/09:  16%|█▌        | 9/56 [00:11<01:18,  1.67s/it]
Masking lower-iono interferograms IW2/09:  18%|█▊        | 10/56 [00:13<01:19,  1.73s/it]
Masking lower-iono interferograms IW1/03:  18%|█▊        | 10/56 [00:13<01:19,  1.73s/it]
Masking lower-iono interferograms IW1/03:  20%|█▉        | 11/56 [00:14<01:08,  1.53s/it]
Masking lower-iono interferograms IW1/08:  20%|█▉        | 11/56 [00:14<01:08,  1.53s/it]
Masking lower-iono interferograms IW1/08:  21%|██�       | 12/56 [00:16<01:13,  1.67s/it]
Masking lower-iono interferograms IW1/02:  21%|██�       | 12/56 [00:16<01:13,  1.67s/it]
Masking lower-iono interferograms IW1/02:  23%|██▎       | 13/56 [00:18<01:07,  1.57s/it]
Masking lower-iono interferograms IW1/04:  23%|██▎       | 13/56 [00:18<01:07,  1.57s/it]
Masking lower-iono interferograms IW1/04:  25%|██▌       | 14/56 [00:19<01:03,  1.50s/it]
Masking lower-iono interferograms IW1/05:  25%|██▌       | 14/56 [00:19<01:03,  1.50s/it]
Masking lower-iono interferograms IW1/05:  27%|██▋       | 15/56 [00:21<01:02,  1.53s/it]
Masking lower-iono interferograms IW1/06:  27%|██▋       | 15/56 [00:21<01:02,  1.53s/it]
Masking lower-iono interferograms IW1/06:  29%|██▊       | 16/56 [00:22<01:04,  1.61s/it]
Masking lower-iono interferograms IW1/01:  29%|██▊       | 16/56 [00:22<01:04,  1.61s/it]
Masking lower-iono interferograms IW1/01:  30%|███       | 17/56 [00:24<01:04,  1.67s/it]
Masking lower-iono interferograms IW1/07:  30%|███       | 17/56 [00:24<01:04,  1.67s/it]
Masking lower-iono interferograms IW1/07:  32%|███�      | 18/56 [00:26<01:04,  1.70s/it]
Masking lower-iono interferograms IW1/09:  32%|███�      | 18/56 [00:26<01:04,  1.70s/it]
Masking lower-iono interferograms IW1/09:  34%|███�      | 19/56 [00:28<01:03,  1.72s/it]
Masking lower-iono interferograms IW3/03:  34%|███�      | 19/56 [00:28<01:03,  1.72s/it]
Masking lower-iono interferograms IW3/03:  36%|███▌      | 20/56 [00:30<01:04,  1.80s/it]
Masking lower-iono interferograms IW3/08:  36%|███▌      | 20/56 [00:30<01:04,  1.80s/it]
Masking lower-iono interferograms IW3/08:  38%|███▊      | 21/56 [00:32<01:06,  1.89s/it]
Masking lower-iono interferograms IW3/02:  38%|███▊      | 21/56 [00:32<01:06,  1.89s/it]
Masking lower-iono interferograms IW3/02:  39%|███▉      | 22/56 [00:34<01:11,  2.09s/it]
Masking lower-iono interferograms IW3/04:  39%|███▉      | 22/56 [00:34<01:11,  2.09s/it]
Masking lower-iono interferograms IW3/04:  41%|████      | 23/56 [00:36<01:08,  2.09s/it]
Masking lower-iono interferograms IW3/05:  41%|████      | 23/56 [00:36<01:08,  2.09s/it]
Masking lower-iono interferograms IW3/05:  43%|████▎     | 24/56 [00:38<01:05,  2.05s/it]
Masking lower-iono interferograms IW3/06:  43%|████▎     | 24/56 [00:38<01:05,  2.05s/it]
Masking lower-iono interferograms IW3/06:  45%|████�     | 25/56 [00:40<01:02,  2.03s/it]
Masking lower-iono interferograms IW3/01:  45%|████�     | 25/56 [00:40<01:02,  2.03s/it]
Masking lower-iono interferograms IW3/01:  46%|████▋     | 26/56 [00:42<01:00,  2.01s/it]
Masking lower-iono interferograms IW3/07:  46%|████▋     | 26/56 [00:42<01:00,  2.01s/it]
Masking lower-iono interferograms IW3/07:  48%|████▊     | 27/56 [00:44<00:57,  1.99s/it]
Masking lower-iono interferograms IW3/09:  48%|████▊     | 27/56 [00:44<00:57,  1.99s/it]
Masking lower-iono interferograms IW3/09:  50%|█████     | 28/56 [00:46<00:55,  1.99s/it]
Masking upper-iono interferograms IW2/10:  50%|█████     | 28/56 [00:46<00:55,  1.99s/it]
Masking upper-iono interferograms IW2/10:  52%|█████�    | 29/56 [00:48<00:54,  2.00s/it]
Masking upper-iono interferograms IW2/03:  52%|█████�    | 29/56 [00:48<00:54,  2.00s/it]
Masking upper-iono interferograms IW2/03:  54%|█████▎    | 30/56 [00:50<00:52,  2.00s/it]
Masking upper-iono interferograms IW2/08:  54%|█████▎    | 30/56 [00:50<00:52,  2.00s/it]
Masking upper-iono interferograms IW2/08:  55%|█████▌    | 31/56 [00:52<00:50,  2.01s/it]
Masking upper-iono interferograms IW2/02:  55%|█████▌    | 31/56 [00:52<00:50,  2.01s/it]
Masking upper-iono interferograms IW2/02:  57%|█████▋    | 32/56 [00:54<00:48,  2.02s/it]
Masking upper-iono interferograms IW2/04:  57%|█████▋    | 32/56 [00:54<00:48,  2.02s/it]
Masking upper-iono interferograms IW2/04:  59%|█████▉    | 33/56 [00:56<00:46,  2.02s/it]
Masking upper-iono interferograms IW2/05:  59%|█████▉    | 33/56 [00:56<00:46,  2.02s/it]
Masking upper-iono interferograms IW2/05:  61%|██████    | 34/56 [00:58<00:44,  2.02s/it]
Masking upper-iono interferograms IW2/06:  61%|██████    | 34/56 [00:58<00:44,  2.02s/it]
Masking upper-iono interferograms IW2/06:  62%|██████▎   | 35/56 [01:00<00:42,  2.02s/it]
Masking upper-iono interferograms IW2/01:  62%|██████▎   | 35/56 [01:00<00:42,  2.02s/it]
Masking upper-iono interferograms IW2/01:  64%|██████�   | 36/56 [01:02<00:40,  2.02s/it]
Masking upper-iono interferograms IW2/07:  64%|██████�   | 36/56 [01:02<00:40,  2.02s/it]
Masking upper-iono interferograms IW2/07:  66%|██████▌   | 37/56 [01:04<00:38,  2.02s/it]
Masking upper-iono interferograms IW2/09:  66%|██████▌   | 37/56 [01:04<00:38,  2.02s/it]
Masking upper-iono interferograms IW2/09:  68%|██████▊   | 38/56 [01:07<00:36,  2.02s/it]
Masking upper-iono interferograms IW1/03:  68%|██████▊   | 38/56 [01:07<00:36,  2.02s/it]
Masking upper-iono interferograms IW1/03:  70%|██████▉   | 39/56 [01:08<00:32,  1.91s/it]
Masking upper-iono interferograms IW1/08:  70%|██████▉   | 39/56 [01:08<00:32,  1.91s/it]
Masking upper-iono interferograms IW1/08:  71%|███████�  | 40/56 [01:10<00:29,  1.84s/it]
Masking upper-iono interferograms IW1/02:  71%|███████�  | 40/56 [01:10<00:29,  1.84s/it]
Masking upper-iono interferograms IW1/02:  73%|███████▎  | 41/56 [01:12<00:27,  1.81s/it]
Masking upper-iono interferograms IW1/04:  73%|███████▎  | 41/56 [01:12<00:27,  1.81s/it]
Masking upper-iono interferograms IW1/04:  75%|███████▌  | 42/56 [01:13<00:24,  1.77s/it]
Masking upper-iono interferograms IW1/05:  75%|███████▌  | 42/56 [01:13<00:24,  1.77s/it]
Masking upper-iono interferograms IW1/05:  77%|███████▋  | 43/56 [01:15<00:22,  1.75s/it]
Masking upper-iono interferograms IW1/06:  77%|███████▋  | 43/56 [01:15<00:22,  1.75s/it]
Masking upper-iono interferograms IW1/06:  79%|███████▊  | 44/56 [01:17<00:20,  1.74s/it]
Masking upper-iono interferograms IW1/01:  79%|███████▊  | 44/56 [01:17<00:20,  1.74s/it]
Masking upper-iono interferograms IW1/01:  80%|████████  | 45/56 [01:18<00:18,  1.73s/it]
Masking upper-iono interferograms IW1/07:  80%|████████  | 45/56 [01:18<00:18,  1.73s/it]
Masking upper-iono interferograms IW1/07:  82%|████████� | 46/56 [01:20<00:17,  1.72s/it]
Masking upper-iono interferograms IW1/09:  82%|████████� | 46/56 [01:20<00:17,  1.72s/it]
Masking upper-iono interferograms IW1/09:  84%|████████� | 47/56 [01:22<00:15,  1.71s/it]
Masking upper-iono interferograms IW3/03:  84%|████████� | 47/56 [01:22<00:15,  1.71s/it]
Masking upper-iono interferograms IW3/03:  86%|████████▌ | 48/56 [01:24<00:14,  1.79s/it]
Masking upper-iono interferograms IW3/08:  86%|████████▌ | 48/56 [01:24<00:14,  1.79s/it]
Masking upper-iono interferograms IW3/08:  88%|████████▊ | 49/56 [01:26<00:12,  1.85s/it]
Masking upper-iono interferograms IW3/02:  88%|████████▊ | 49/56 [01:26<00:12,  1.85s/it]
Masking upper-iono interferograms IW3/02:  89%|████████▉ | 50/56 [01:28<00:11,  1.88s/it]
Masking upper-iono interferograms IW3/04:  89%|████████▉ | 50/56 [01:28<00:11,  1.88s/it]
Masking upper-iono interferograms IW3/04:  91%|█████████ | 51/56 [01:30<00:09,  1.90s/it]
Masking upper-iono interferograms IW3/05:  91%|█████████ | 51/56 [01:30<00:09,  1.90s/it]
Masking upper-iono interferograms IW3/05:  93%|█████████▎| 52/56 [01:32<00:07,  1.93s/it]
Masking upper-iono interferograms IW3/06:  93%|█████████▎| 52/56 [01:32<00:07,  1.93s/it]
Masking upper-iono interferograms IW3/06:  95%|█████████�| 53/56 [01:34<00:05,  1.94s/it]
Masking upper-iono interferograms IW3/01:  95%|█████████�| 53/56 [01:34<00:05,  1.94s/it]
Masking upper-iono interferograms IW3/01:  96%|█████████▋| 54/56 [01:36<00:03,  1.94s/it]
Masking upper-iono interferograms IW3/07:  96%|█████████▋| 54/56 [01:36<00:03,  1.94s/it]
Masking upper-iono interferograms IW3/07:  98%|█████████▊| 55/56 [01:37<00:01,  1.95s/it]
Masking upper-iono interferograms IW3/09:  98%|█████████▊| 55/56 [01:37<00:01,  1.95s/it]
Masking upper-iono interferograms IW3/09: 100%|██████████| 56/56 [01:39<00:00,  1.96s/it]
Masking upper-iono interferograms IW3/09: 100%|██████████| 56/56 [01:39<00:00,  1.78s/it]
after adjustment (index in merged image, index starts from 1): 
 burst     fl      ll      fc      lc         fl      ll      fc      lc         fl      ll      fc      lc       
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
   1         51    1450     601   20600        501    1900   20601   44400       2301    3750   44401   67200    
   2       1451    2800     601   20600       1901    3250   20601   44400       3751    5100   44401   67200    
   3       2801    4150     601   20600       3251    4600   20601   44400       5101    6400   44401   67200    
   4       4151    5500     601   20600       4601    5950   20601   44400       6401    7750   44401   67200    
   5       5501    6850     601   20600       5951    7300   20601   44400       7751    9100   44401   67200    
   6       6851    8150     601   20600       7301    8650   20601   44400       9101   10450   44401   67200    
   7       8151    9500     401   20600       8651   10000   20601   44400      10451   11800   44401   67200    
   8       9501   10850     401   20600      10001   11300   20601   44400      11801   13150   44401   67200    
   9      10851   12200     401   20600      11301   12650   20601   44400      13151   14450   44401   67200    
  10      12651   14000   20201   44400    
Multilooking ion/lower/merged/topophase.flat.full ...
GDAL open (R): ion/lower/merged/topophase.flat.full.vrt
API open (WR): ion/lower/merged/topophase.flat
GDAL close: ion/lower/merged/topophase.flat.full.vrt
API close:  ion/lower/merged/topophase.flat
API open (R): ion/lower/merged/topophase.flat
API close:  ion/lower/merged/topophase.flat
Writing geotrans to VRT for ion/lower/merged/topophase.flat
after adjustment (index in merged image, index starts from 1): 
 burst     fl      ll      fc      lc         fl      ll      fc      lc         fl      ll      fc      lc       
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
   1         51    1450     601   20600        501    1900   20601   44400       2301    3750   44401   67200    
   2       1451    2800     601   20600       1901    3250   20601   44400       3751    5100   44401   67200    
   3       2801    4150     601   20600       3251    4600   20601   44400       5101    6400   44401   67200    
   4       4151    5500     601   20600       4601    5950   20601   44400       6401    7750   44401   67200    
   5       5501    6850     601   20600       5951    7300   20601   44400       7751    9100   44401   67200    
   6       6851    8150     601   20600       7301    8650   20601   44400       9101   10450   44401   67200    
   7       8151    9500     401   20600       8651   10000   20601   44400      10451   11800   44401   67200    
   8       9501   10850     401   20600      10001   11300   20601   44400      11801   13150   44401   67200    
   9      10851   12200     401   20600      11301   12650   20601   44400      13151   14450   44401   67200    
  10      12651   14000   20201   44400    
Multilooking ion/upper/merged/topophase.flat.full ...
GDAL open (R): ion/upper/merged/topophase.flat.full.vrt
API open (WR): ion/upper/merged/topophase.flat
GDAL close: ion/upper/merged/topophase.flat.full.vrt
API close:  ion/upper/merged/topophase.flat
API open (R): ion/upper/merged/topophase.flat
API close:  ion/upper/merged/topophase.flat
Writing geotrans to VRT for ion/upper/merged/topophase.flat
unwrapping lower and upper band interferograms
snaphu v1.4.2
Reading wrapped phase from file ion/lower/merged/topophase.flat
No weight file specified.  Assuming uniform weights
Reading correlation data from file ion/lower/merged/topophase.cor
Calculating smooth-solution cost parameters
Initializing flows with MCF algorithm
Integrating phase
Writing output to file ion/lower/merged/filt_topophase.unw
Growing connected components on second pass
snaphu v1.4.2
Reading unwrapped phase from file ion/lower/merged/filt_topophase.unw
No weight file specified.  Assuming uniform weights
Reading correlation data from file ion/lower/merged/topophase.cor
Calculating smooth-solution cost parameters
Growing connected component mask
Writing connected components to file ion/lower/merged/filt_topophase.unw.conncomp
API open (R): ion/lower/merged/filt_topophase.unw
API close:  ion/lower/merged/filt_topophase.unw
GDAL open (R): ion/lower/merged/filt_topophase.unw.vrt
GDAL close: ion/lower/merged/filt_topophase.unw.vrt
API open (R): ion/lower/merged/filt_topophase.unw.conncomp
API close:  ion/lower/merged/filt_topophase.unw.conncomp
GDAL open (R): ion/lower/merged/filt_topophase.unw.conncomp.vrt
GDAL close: ion/lower/merged/filt_topophase.unw.conncomp.vrt
imageMath.py -e='a_0*(abs(b)!=0);a_1*(abs(b)!=0)' --a=tmp.unw --b=ion/lower/merged/topophase.flat -s BIL -o=ion/lower/merged/filt_topophase.unw
GDAL open (R): ion/lower/merged/filt_topophase.unw.vrt
GDAL close: ion/lower/merged/filt_topophase.unw.vrt
This is the Open Source version of ISCE.
Some of the workflows depend on a separate licensed package.
To obtain the licensed package, please make a request for ISCE
through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.
Alternatively, if you are a member, or can become a member of WinSAR
you may be able to obtain access to a version of the licensed sofware at
https://winsar.unavco.org/software/isce
args:  Namespace(hh=None, equation='a_0*(abs(b)!=0);a_1*(abs(b)!=0)', out='ion/lower/merged/filt_topophase.unw', scheme='BIL', dtype='float', debug=False, noxml=False)
files:  ['--a=tmp.unw', '--b=ion/lower/merged/topophase.flat']
Bounding boxes dont match. Not adding bbox info.
snaphu v1.4.2
Reading wrapped phase from file ion/upper/merged/topophase.flat
No weight file specified.  Assuming uniform weights
Reading correlation data from file ion/lower/merged/topophase.cor
Calculating smooth-solution cost parameters
Initializing flows with MCF algorithm
Integrating phase
Writing output to file ion/upper/merged/filt_topophase.unw
Growing connected components on second pass
snaphu v1.4.2
Reading unwrapped phase from file ion/upper/merged/filt_topophase.unw
No weight file specified.  Assuming uniform weights
Reading correlation data from file ion/lower/merged/topophase.cor
Calculating smooth-solution cost parameters
Growing connected component mask
Writing connected components to file ion/upper/merged/filt_topophase.unw.conncomp
API open (R): ion/upper/merged/filt_topophase.unw
API close:  ion/upper/merged/filt_topophase.unw
GDAL open (R): ion/upper/merged/filt_topophase.unw.vrt
GDAL close: ion/upper/merged/filt_topophase.unw.vrt
API open (R): ion/upper/merged/filt_topophase.unw.conncomp
API close:  ion/upper/merged/filt_topophase.unw.conncomp
GDAL open (R): ion/upper/merged/filt_topophase.unw.conncomp.vrt
GDAL close: ion/upper/merged/filt_topophase.unw.conncomp.vrt
imageMath.py -e='a_0*(abs(b)!=0);a_1*(abs(b)!=0)' --a=tmp.unw --b=ion/upper/merged/topophase.flat -s BIL -o=ion/upper/merged/filt_topophase.unw
GDAL open (R): ion/upper/merged/filt_topophase.unw.vrt
GDAL close: ion/upper/merged/filt_topophase.unw.vrt
This is the Open Source version of ISCE.
Some of the workflows depend on a separate licensed package.
To obtain the licensed package, please make a request for ISCE
through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.
Alternatively, if you are a member, or can become a member of WinSAR
you may be able to obtain access to a version of the licensed sofware at
https://winsar.unavco.org/software/isce
args:  Namespace(hh=None, equation='a_0*(abs(b)!=0);a_1*(abs(b)!=0)', out='ion/upper/merged/filt_topophase.unw', scheme='BIL', dtype='float', debug=False, noxml=False)
files:  ['--a=tmp.unw', '--b=ion/upper/merged/topophase.flat']
Bounding boxes dont match. Not adding bbox info.
Do bridging ion/lower/merged/filt_topophase.unw
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
imageMath.py -e='a_0*(abs(b)!=0);a_1*(abs(b)!=0)' --a=tmp.unw --b=ion/lower/merged/filt_topophase.unw.conncomp -s BIL -o=ion/lower/merged/filt_topophase.unw
GDAL open (R): ion/lower/merged/filt_topophase.unw.vrt
GDAL close: ion/lower/merged/filt_topophase.unw.vrt
This is the Open Source version of ISCE.
Some of the workflows depend on a separate licensed package.
To obtain the licensed package, please make a request for ISCE
through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.
Alternatively, if you are a member, or can become a member of WinSAR
you may be able to obtain access to a version of the licensed sofware at
https://winsar.unavco.org/software/isce
args:  Namespace(hh=None, equation='a_0*(abs(b)!=0);a_1*(abs(b)!=0)', out='ion/lower/merged/filt_topophase.unw', scheme='BIL', dtype='float', debug=False, noxml=False)
files:  ['--a=tmp.unw', '--b=ion/lower/merged/filt_topophase.unw.conncomp']
Do bridging ion/upper/merged/filt_topophase.unw
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
Deramp
imageMath.py -e='a_0*(abs(b)!=0);a_1*(abs(b)!=0)' --a=tmp.unw --b=ion/upper/merged/filt_topophase.unw.conncomp -s BIL -o=ion/upper/merged/filt_topophase.unw
GDAL open (R): ion/upper/merged/filt_topophase.unw.vrt
GDAL close: ion/upper/merged/filt_topophase.unw.vrt
This is the Open Source version of ISCE.
Some of the workflows depend on a separate licensed package.
To obtain the licensed package, please make a request for ISCE
through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.
Alternatively, if you are a member, or can become a member of WinSAR
you may be able to obtain access to a version of the licensed sofware at
https://winsar.unavco.org/software/isce
args:  Namespace(hh=None, equation='a_0*(abs(b)!=0);a_1*(abs(b)!=0)', out='ion/upper/merged/filt_topophase.unw', scheme='BIL', dtype='float', debug=False, noxml=False)
files:  ['--a=tmp.unw', '--b=ion/upper/merged/filt_topophase.unw.conncomp']
Writing geotrans to VRT for ion/lower/merged/40rlks_10alks/topophase.flat
Writing geotrans to VRT for ion/upper/merged/40rlks_10alks/topophase.flat
Multilooking ion/lower/merged/40rlks_10alks/topophase.flat ...
GDAL open (R): ion/lower/merged/40rlks_10alks/topophase.flat.vrt
API open (WR): ion/lower/merged/topophase.flat
GDAL close: ion/lower/merged/40rlks_10alks/topophase.flat.vrt
API close:  ion/lower/merged/topophase.flat
API open (R): ion/lower/merged/topophase.flat
API close:  ion/lower/merged/topophase.flat
Writing geotrans to VRT for ion/lower/merged/topophase.flat
Multilooking ion/upper/merged/40rlks_10alks/topophase.flat ...
GDAL open (R): ion/upper/merged/40rlks_10alks/topophase.flat.vrt
API open (WR): ion/upper/merged/topophase.flat
GDAL close: ion/upper/merged/40rlks_10alks/topophase.flat.vrt
API close:  ion/upper/merged/topophase.flat
API open (R): ion/upper/merged/topophase.flat
API close:  ion/upper/merged/topophase.flat
Writing geotrans to VRT for ion/upper/merged/topophase.flat
computing ionosphere
after adjustment:
Traceback (most recent call last):
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 352, in <module>
    main()
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 348, in main
    sys.exit(process_entry_point.load()())
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 207, in gunw_slc
    iono_processing(
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/iono_proc.py", line 118, in iono_processing
    runIon.ionosphere(topsapp, ionParam)
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce/components/isceobj/TopsProc/runIon.py", line 1308, in ionosphere
    ionos = computeIonosphere(lowerUnw, upperUnw, cor, fl, fu, adjFlag, corThresholdAdj, 0)
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce/components/isceobj/TopsProc/runIon.py", line 1250, in computeIonosphere
    print('max phase difference: {}'.format(np.amax(unw_diff)))
  File "<__array_function__ internals>", line 180, in amax
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 2793, in amax
    return _wrapreduction(a, np.maximum, 'max', axis, None, out,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
ValueError: zero-size array to reduction operation maximum which has no identity

All the failed jobs are here:

{'expiration_time': '2023-11-22T00:00:00+00:00',
 'job_id': 'b9a37165-8c1c-4104-a52a-99e6bcd28a2a',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 20342,
                    'granules': ['S1A_IW_SLC__1SDV_20220210T162911_20220210T162938_041853_04FB8A_974D',
                                 'S1A_IW_SLC__1SDV_20220210T162936_20220210T163004_041853_04FB8A_4E1F'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20220117T162912_20220117T162939_041503_04EF80_77A2',
                                           'S1A_IW_SLC__1SDV_20220117T162937_20220117T163004_041503_04EF80_0BFA'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE_TEST',
 'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/b9a37165-8c1c-4104-a52a-99e6bcd28a2a/b9a37165-8c1c-4104-a52a-99e6bcd28a2a.log'],
 'name': 'Anchorage_131_HRRR_0524',
 'processing_times': [9269.375],
 'request_time': '2023-05-24T23:49:30+00:00',
 'status_code': 'FAILED',
 'user_id': 'cmarshak'}
{'expiration_time': '2023-11-22T00:00:00+00:00',
 'job_id': 'de0e0133-39b4-4bc1-9c44-352bbdb400a8',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 20342,
                    'granules': ['S1A_IW_SLC__1SDV_20220129T162911_20220129T162938_041678_04F56E_51C5',
                                 'S1A_IW_SLC__1SDV_20220129T162936_20220129T163004_041678_04F56E_60B3'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20220117T162912_20220117T162939_041503_04EF80_77A2',
                                           'S1A_IW_SLC__1SDV_20220117T162937_20220117T163004_041503_04EF80_0BFA'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE_TEST',
 'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/de0e0133-39b4-4bc1-9c44-352bbdb400a8/de0e0133-39b4-4bc1-9c44-352bbdb400a8.log'],
 'name': 'Anchorage_131_HRRR_0524',
 'processing_times': [9132.298],
 'request_time': '2023-05-24T23:49:30+00:00',
 'status_code': 'FAILED',
 'user_id': 'cmarshak'}
{'expiration_time': '2023-11-22T00:00:00+00:00',
 'job_id': '212c21e5-61df-4f58-aabc-84f437e3ab76',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 20342,
                    'granules': ['S1A_IW_SLC__1SDV_20220210T162911_20220210T162938_041853_04FB8A_974D',
                                 'S1A_IW_SLC__1SDV_20220210T162936_20220210T163004_041853_04FB8A_4E1F'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20220129T162911_20220129T162938_041678_04F56E_51C5',
                                           'S1A_IW_SLC__1SDV_20220129T162936_20220129T163004_041678_04F56E_60B3'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE_TEST',
 'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/212c21e5-61df-4f58-aabc-84f437e3ab76/212c21e5-61df-4f58-aabc-84f437e3ab76.log'],
 'name': 'Anchorage_131_HRRR_0524',
 'processing_times': [9044.308],
 'request_time': '2023-05-24T23:49:30+00:00',
 'status_code': 'FAILED',
 'user_id': 'cmarshak'}
{'expiration_time': '2023-11-22T00:00:00+00:00',
 'job_id': '781c2c86-a6df-4617-bcb1-73bfa73a49c8',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 20342,
                    'granules': ['S1A_IW_SLC__1SDV_20220222T162911_20220222T162938_042028_0501A2_0E39',
                                 'S1A_IW_SLC__1SDV_20220222T162936_20220222T163004_042028_0501A2_8FED'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20220129T162911_20220129T162938_041678_04F56E_51C5',
                                           'S1A_IW_SLC__1SDV_20220129T162936_20220129T163004_041678_04F56E_60B3'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE_TEST',
 'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/781c2c86-a6df-4617-bcb1-73bfa73a49c8/781c2c86-a6df-4617-bcb1-73bfa73a49c8.log'],
 'name': 'Anchorage_131_HRRR_0524',
 'processing_times': [9114.568],
 'request_time': '2023-05-24T23:49:30+00:00',
 'status_code': 'FAILED',
 'user_id': 'cmarshak'}
{'expiration_time': '2023-11-22T00:00:00+00:00',
 'job_id': 'b81616b8-26b0-4b7a-9cd0-ee4484371768',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 20342,
                    'granules': ['S1A_IW_SLC__1SDV_20220222T162911_20220222T162938_042028_0501A2_0E39',
                                 'S1A_IW_SLC__1SDV_20220222T162936_20220222T163004_042028_0501A2_8FED'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20220210T162911_20220210T162938_041853_04FB8A_974D',
                                           'S1A_IW_SLC__1SDV_20220210T162936_20220210T163004_041853_04FB8A_4E1F'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE_TEST',
 'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/b81616b8-26b0-4b7a-9cd0-ee4484371768/b81616b8-26b0-4b7a-9cd0-ee4484371768.log'],
 'name': 'Anchorage_131_HRRR_0524',
 'processing_times': [9218.873],
 'request_time': '2023-05-24T23:49:30+00:00',
 'status_code': 'FAILED',
 'user_id': 'cmarshak'}

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions