Need help for exporting ISCE --> dolphin --> mintpy

[Karthi-Wolf]

Hello everyone,
I'm new to the dolphin. I very much interested to do the PD/DS insar using it. I got error while starting mintpy from the output of ISCE into dolphin. ( I coregistered using ISCE2 and ran dolphin then I trying to export into mintpy.
Your help will be greatly appreciated.
Here this image shows mintpy config file setup i did

Here the error I encountered.

create HDF5 file /home/incois/mohanty/karthi/SBAS/dolphin/mintpy_1/inputs/ifgramStack.h5 with w mode
create dataset /unwrapPhase      of <class 'numpy.float32'>   in size of (765, 4167, 20527) with compression = None
[==================================================] 20241009_20241021 1305s /   230s
create dataset /coherence        of <class 'numpy.float32'>   in size of (765, 4167, 20527) with compression = None
[==================================================] 20241009_20241021 7366s /   150s
create dataset /connectComponent of <class 'numpy.int16'>     in size of (765, 4167, 20527) with compression = lzf
[==================================================] 20241009_20241021 4187s /    85s
create dataset /date             of <class 'numpy.bytes_'>    in size of (765, 2)
create dataset /bperp            of <class 'numpy.float32'>   in size of (765,)
create dataset /dropIfgram       of <class 'numpy.bool_'>     in size of (765,)
add extra metadata: {'PROJECT_NAME': 'mintpy'}
Finished writing to /home/incois/mohanty/karthi/SBAS/dolphin/mintpy_1/inputs/ifgramStack.h5
--------------------------------------------------
searching ionosphere pairs info
input data files:
WARNING: No data files found for the required dataset: ['unwrapPhase']! Skip loading for ionosphere stack.
--------------------------------------------------
searching offset pairs info
input data files:
WARNING: No data files found for the required dataset: ['rangeOffset', 'azimuthOffset']! Skip loading for offset stack.
time used: 390 mins 38.4 secs.

Traceback (most recent call last):
 File "/home/incois/mohanty/.conda/envs/isce2/bin/smallbaselineApp.py", line 10, in <module>
   sys.exit(main())
 File "/home/incois/mohanty/.conda/envs/isce2/lib/python3.9/site-packages/mintpy/cli/smallbaselineApp.py", line 209, in main
   run_smallbaselineApp(inps)
 File "/home/incois/mohanty/.conda/envs/isce2/lib/python3.9/site-packages/mintpy/smallbaselineApp.py", line 1155, in run_smallbaselineApp
   app.run(steps=inps.runSteps)
 File "/home/incois/mohanty/.conda/envs/isce2/lib/python3.9/site-packages/mintpy/smallbaselineApp.py", line 908, in run
   self.run_load_data(sname)
 File "/home/incois/mohanty/.conda/envs/isce2/lib/python3.9/site-packages/mintpy/smallbaselineApp.py", line 182, in run_load_data
   stack_file, geom_file, _, ion_file = ut.check_loaded_dataset(self.workDir, print_msg=True)[:4]
 File "/home/incois/mohanty/.conda/envs/isce2/lib/python3.9/site-packages/mintpy/utils/utils.py", line 91, in check_loaded_dataset
   raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), geom_file + msg_aria)
FileNotFoundError: [Errno 2] No such file or directory: '/home/incois/mohanty/karthi/SBAS/dolphin/mintpy_1/inputs/geometryGeo.h5'" 

Detailed full log of 'smallbaselineApp.py mintpy.txt' as txt file

dolphin error HPC.txt

[scottstanie]

Can you describe what input data you are processing, and the other steps you ran to get to this point? it says it's looking for a geometryGeo.h5 file; did you process in geo coordinates or radar coordinates?

[Karthi-Wolf]

Yes. Here what I followed.
I made coregistered SLCs usiing
"stackSentinel.py -s/..../slc -o /..../orbits -d /.../dem/demLat_N19_N22_Lon_E084_E087.dem.wgs84 -b '19.760 19.917 85.725 85.935' -a /..../orbits -c 5 -C NESD -W slc -m 20160101 -e 0.30"

Then I ran dolphin by 'dolphin_config.yaml'
It was like this
'```

REQUIRED: list of CSLC files, or newline-delimited file containing list of CSLC files.

Type: array.

cslc_file_list:

• /....../coregistered/merged/SLC/20160101/20160101.slc.full.vrt
• /....../coregistered/merged/SLC/20160113/20160113.slc.full.vrt
• /....../coregistered/merged/SLC/20160125/20160125.slc.full.vrt
• /....../coregistered/merged/SLC/20160206/20160206.slc.full.vrt
• /....../coregistered/merged/SLC/20160218/20160218.slc.full.vrt
• ...
• ...
• ...'

  I got dolphin directory. It's structure looks likes (tree) 
[dolphin directory tree.txt](https://github.com/user-attachments/files/24900023/dolphin.directory.tree.txt)

I then directly try to import it in the mintpy by indicting the path like I shown above.
And I did like this,,, \
"```
##-------------------------------- MintPy -----------------------------##
########## 1. Load Data (--load to exit after this step)
## load_data.py -H to check more details and example inputs.
mintpy.load.processor        = isce
##---------for ISCE only:
mintpy.load.metaFile         = /...../coregistered/reference/IW*.xml
mintpy.load.baselineDir      = /...../coregistered/baselines
##---------interferogram datasets:
mintpy.load.unwFile          = /...../dolphin/unwrapped/*.unw.tif
mintpy.load.corFile          = /...../dolphin/interferograms/*.int.cor.tif
mintpy.load.connCompFile     = /...../dolphin/unwrapped/*.unw.conncomp.tif
##---------geometry datasets:
mintpy.load.demFile          = /...../coregistered/merged/geom_reference/hgt.rdr
mintpy.load.lookupYFile      = /...../coregistered/merged/geom_reference/lat.rdr
mintpy.load.lookupXFile      = /...../coregistered/merged/geom_reference/lon.rdr
mintpy.load.incAngleFile     = /...../coregistered/merged/geom_reference/los.rdr
mintpy.load.azAngleFile      = /...../coregistered/merged/geom_reference/los.rdr
mintpy.load.shadowMaskFile   = /...../coregistered/merged/geom_reference/shadowMask.rdr
mintpy.load.waterMaskFile    = /...../coregistered/merged/geom_reference/waterMask.rdr

"
Am I missing anything.
Thank you...

[Karthi-Wolf]

In short I have processed in radar coordinates. Can you please help What further things i need to follow?

[giorgoskouroupis]

Hello all :)

If I may say so, among others, one difference between dolphin and MintPy data is in their metadata content. The h5 data contain a lot of metadata which are not present in the geotifs of dolphin, and even worse, dolphin could contain metadata that would be misinterpreted from MintPy.

A simple info.py /path/to/dolphin_scene.tif will reveal to you the metadata of dolphin. take a look at the usage examples: https://github.com/insarlab/MintPy/blob/85e3ae888fbf4a5fded369e48fa8d6edc9c9866e/src/mintpy/cli/info.py#L15-L40

MintPy will implicitly check whether a scene is geocoded or not based on such metadata whilst trying to load the data: https://github.com/insarlab/MintPy/blob/85e3ae888fbf4a5fded369e48fa8d6edc9c9866e/src/mintpy/load_data.py#L494-L499

In your case, my guess is that you will observe that your dolphin timeseries tifs contain metadata such as Y_FIRST that will make MintPy think that this corresponds to a geocoded scene while in effect it is in radar coordinates, hence the missing geometry file in geo-coordinates.

My suggestion would be to make the metadata of the dolphin outcome be as they are required by MintPy to drive the correct process. you can take inspiration from: https://github.com/insarlab/MintPy/blob/main/src/mintpy/objects/stack.py or you could simply make a dummy ISCE2-MintPy run and check what kind of data are acceptable by MintPy and the processes you to do and try to recreate them.

[Karthi-Wolf]

Hi everyone!
@giorgoskouroupis @scottstanie Thanks for the clarifications.

I have seen the recent geocoding options in dolphin and pretty excited with the ease of use. Its a great access to process with the builtin feature.

I have ran ISCE2 --> Dolphin --> Mintpy finally.
I have also enquired about my problems in the #506 comment

I have done dolphin geocode with the strides (x,y) (9,3)

I was trying to make it Minty friendly and it worked finally. I have used this geocodeUsingGdalWarp() funtion to the ISCE2 geometries.

def geocodeUsingGdalWarp(infile, latfile, lonfile, outfile,
                         insrs=4326, outsrs=None,
                         spacing=None, fmt='GTiff', bounds=None,
                         method='near'):
    '''
    Geocode a swath file using corresponding lat, lon files
    '''
    sourcexmltmpl = '''    <SimpleSource>
      <SourceFilename>{0}</SourceFilename>
      <SourceBand>{1}</SourceBand>
    </SimpleSource>'''
    
    driver = gdal.GetDriverByName('VRT')
    tempvrtname = 'geocode.vrt'
    inds = gdal.OpenShared(infile, gdal.GA_ReadOnly)
    
    tempds = driver.Create(tempvrtname, inds.RasterXSize, inds.RasterYSize, 0)
    
    for ii in range(inds.RasterCount):
        band = inds.GetRasterBand(1)
        tempds.AddBand(band.DataType)
        tempds.GetRasterBand(ii+1).SetMetadata({'source_0': sourcexmltmpl.format(infile, ii+1)}, 'vrt_sources')
  
    sref = osr.SpatialReference()
    sref.ImportFromEPSG(insrs)
    srswkt = sref.ExportToWkt()

    tempds.SetMetadata({'SRS' : srswkt,
                        'X_DATASET': lonfile,
                        'X_BAND' : '1',
                        'Y_DATASET': latfile,
                        'Y_BAND' : '1',
                        'PIXEL_OFFSET' : '0',
                        'LINE_OFFSET' : '0',
                        'PIXEL_STEP' : '1',
                        'LINE_STEP' : '1'}, 'GEOLOCATION')
    
    band = None
    tempds = None 
    inds = None
    
    if spacing is None:
        spacing = [None, None]
    
    warpOptions = gdal.WarpOptions(format=fmt,
                                   xRes=spacing[0], yRes=spacing[0],
                                   dstSRS=outsrs, outputBounds = bounds, 
                                   resampleAlg=method, geoloc=True)
    gdal.Warp(outfile, tempvrtname, options=warpOptions)

I have taken the corner coordinates and pixel_spacing using the $gdalinfo *.unw.tif (geocoded dolphin file)

$ gdalinfo 20181204_20181228.unw.tif

Driver: GTiff/GeoTIFF
Files: 20181204_20181228.unw.tif
Size is 2475, 1805
Coordinate System is:
GEOGCRS["WGS 84",

Pixel Size = (0.000369012526897,-0.000369012526897)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (  85.3961550,  20.2291355) ( 85d23'46.16"E, 20d13'44.89"N)
Lower Left  (  85.3961550,  19.5630679) ( 85d23'46.16"E, 19d33'47.04"N)
Upper Right (  86.3094611,  20.2291355) ( 86d18'34.06"E, 20d13'44.89"N)
Lower Right (  86.3094611,  19.5630679) ( 86d18'34.06"E, 19d33'47.04"N)
Center      (  85.8528080,  19.8961017) ( 85d51'10.11"E, 19d53'45.97"N)
Band 1 Block=512x512 Type=Float32, ColorInterp=Gray

os.makedirs('strided_geoms',exist_ok=True)
!gdal_translate -outsize 2475 1805 /xx/coregistered/merged/geom_reference/lon.rdr.full.vrt /xx/coregistered/dolphin/gecode_f1/strided_geoms/lon.strided.vrt
!gdal_translate -outsize 2475 1805 /xx/coregistered/merged/geom_reference/lat.rdr.full.vrt /xx/coregistered/dolphin/gecode_f1/strided_geoms/lat.strided.vrt

**Since my geocoded dimensions are (2475, 1805) **
I have already translated lon.rdr.full.vrt & lat.rdr.full.vrt using same dimensions as strided.
I then did the same to the los, hgt, shadowMask (ex below).

!gdal_translate \
-outsize 2475 1805 \
/xx/coregistered/merged/geom_reference/shadowMask.rdr.full.vrt \
/xx/coregistered/dolphin/gecode_f1/strided_geoms/shadowMask.strided.vrt

I then geocoded using geocodeUsingGdalWarp() with the bounds and spacing from the geocoded *.unw.tif file.

bounds = [85.396155047081564, 19.5630679,
          86.3094611, 20.229135497870015]

spacing = [0.000369012526897, 0.000369012526897]


geocodeUsingGdalWarp(
    infile='/xx/coregistered/dolphin/gecode_f1/strided_geoms/los.strided.vrt',
    latfile='/xx/coregistered/dolphin/gecode_f1/strided_geoms/lat.strided.vrt',
    lonfile='/xx/coregistered/dolphin/gecode_f1/strided_geoms/lon.strided.vrt',
    outfile='/xx/coregistered/dolphin/gecode_f1/strided_geoms/los.tif',
    bounds=bounds,
    spacing=spacing,
    outsrs='EPSG:4326'
)

I have extracted band1 incAngle.tif & band 2 azAngle.tif from los.tif

!gdal_translate -b 1 los.tif  incAngle.tif
!gdal_translate -b 2 los.tif azAngle.tif

I gave inputs from the these to Mintpy like below

mintpy.load.processor        = isce
##---------for ISCE only:
mintpy.load.metaFile         = /xx/coregistered/reference/IW*.xml
mintpy.load.baselineDir      = /xx/coregistered/baselines
##---------interferogram datasets:
mintpy.load.unwFile          = /xx/coregistered/dolphin/gecode_f1/unwrapped/*.unw.tif
mintpy.load.corFile          = /xx/coregistered/dolphin/gecode_f1/interferograms/*.int.cor.tif
mintpy.load.connCompFile     = /xx/coregistered/dolphin/gecode_f1/interferograms/*.unw.conncomp.tif
##---------geometry datasets:
mintpy.load.demFile          = /xx/coregistered/dolphin/gecode_f1/strided_geoms/hgt.tif
mintpy.load.lookupYFile      = auto
mintpy.load.lookupXFile      = auto
mintpy.load.incAngleFile     = /xx/coregistered/dolphin/gecode_f1/strided_geoms/incAngle.tif
mintpy.load.azAngleFile      = /xx/coregistered/dolphin/gecode_f1/strided_geoms/azAngle.tif
mintpy.load.shadowMaskFile   = /xx/coregistered/dolphin/gecode_f1/strided_geoms/shadowMask.tif

The result seems to be promising. Though I didn't provide the watermask, It seems it ran well. I need to provide watermask and tropospheric correction yet. It's with the temporal coherence of 0.5, It has more coherence pixels than my previous ISCE2 --> Mintpy.
Can you please look and share your insight on my result. Any comments and ideas will be greatly appreciated.

Regards,
Karthi

[bcankara]

https://github.com/bcankara/Dolphin2MintPy try