Explanation of what is in facetselfcal parmdb's (for getting dtec's for inversion)

[Joshuaalbert]

I am looking for the final merged phase solutions for each facet.
These presumably have not been separated into common scalar phase + delay(nu) + dtec(1/nu) terms.
I find in each facet folder the following parmdbs for e.g.:

L137859_SBgr000-10_uv.dppp.dpppaverage_chunk12_12299B18Ft_0g.convert_merged_selfcal_parmdbs/
L137859_SBgr000-10_uv.dppp.dpppaverage_chunk12_12299B18Ft_0g.merge_amp_parmdbs1/
L137859_SBgr000-10_uv.dppp.dpppaverage_chunk12_12299B18Ft_0g.merge_amp_parmdbs2/
L137859_SBgr000-10_uv.dppp.dpppaverage_chunk12_12299B18Ft_0g.merge_first_phase_parmdbs/
L137859_SBgr000-10_uv.dppp.dpppaverage_chunk12_12299B18Ft_0g.merge_phase_parmdbs/
L137859_SBgr000-10_uv.dppp.dpppaverage_chunk12_12299B18Ft_0g.merge_selfcal_parmdbs/
L137859_SBgr000-10_uv.dppp.dpppaverage_chunk12_12299B18Ft_0g.smooth_amp1/
L137859_SBgr000-10_uv.dppp.dpppaverage_chunk12_12299B18Ft_0g.smooth_amp2/

I'm guessing what I'm looking for is in one of:

L137859_SBgr000-10_uv.dppp.dpppaverage_chunk12_12299B18Ft_0g.convert_merged_selfcal_parmdbs/
L137859_SBgr000-10_uv.dppp.dpppaverage_chunk12_12299B18Ft_0g.merge_phase_parmdbs/
L137859_SBgr000-10_uv.dppp.dpppaverage_chunk12_12299B18Ft_0g.merge_selfcal_parmdbs/

Each one contains a set of rows (one for each antenna+gainX:Y:phase/amp) and each row appears to have StartX, EndX (freqs) and StartY, EndY (times), and Values (shape [numFreq, numTime]).
or is it [numTime, numFreq] ?

What are difference between the parmdb's?

I need the phase containing physical information of common scalar phase + delay(nu) + dtec(1/nu) terms so that the dtec that I input to my ionosphere inversion represents the actual propagation effects.

FYI, I will write a converter to hdf5.

[darafferty]

The merge_selfcal_parmdbs one contains the TEC, CommonScalarPhase, and Gain solutions (so all the selfcal solutions) and so is probably what you want (merge_phase_parmdbs is just TEC and CommonScalarPhase, convert_merged_selfcal_parmdbs is everything in merge_selfcal_parmdbs converted to a single gain solution).

[Joshuaalbert]

Ah okay, so to clarify, which one comes first: convert_merged_selfcal_parmdbs from which common scalar, tec, and delay are separated to the pieces in merge_selfcal_parmdbs or are the by products of calibration the common scalar, tec, and delay from which the single gain is created? I'm concerned about the what TEC is solved for. I would like to look pretty closely at the solver so that I can get an idea of the measurement errors of TEC.

[darafferty]

merge_selfcal_parmdbs is (basically) the output of the solver, so I think this is what you want (the single gain parmdb is created from this). The TEC+CSp solutions are directly from DPPP GainCal (i.e., caltype = tecandphase).

[Joshuaalbert]

Thanks

[Joshuaalbert]

Hey David,

I need another point of clarification, and this seems the best to reopen this.
The name of the solution file, L137859_SBgr000-10_uv.dppp.dpppaverage_chunk12_12299B18Ft_0g.merge_selfcal_parmdbs, seems to imply that it contains only the SBs 000 - 009 and only chunk 12 (in time). Is this just a naming convention where the name is just the first SB block and the last chunk?

Also, in calculating the convert_* gains which frequencies are used? I have basically just startFreq and endFreq and then numFreq, so I would expect the frequencies should be f_n = startfreq + n * 1/2 (endFreq - startFreq)/numFreq with n = 1...numFreq, though I imagine one could have used startFreq + n * (endFreq - startFreq)/numFreq with n = 0...numFreq-1.

[Joshuaalbert]

Actually, I have a more important question here. If I look inside a merge_selfcal_parmdbs table I have rows that are blocked in subband over all time. I.e. something like this (ignoring CS values):

TEC:ANT1 | SB000-SB009 | timestamps 1 to 480 | values
TEC:ANT2 | SB000-SB009 | timestamps 1 to 480 | values
...
TEC:ANTN | SB000-SB009 | timestamps 1 to 480 | values
-----
TEC:ANT1 | SB010-SB019 | timestamps 480 to 960 | values
TEC:ANT2 | SB010-SB019 | timestamps 480 to 960 | values
...
TEC:ANTN | SB010-SB019 | timestamps 480 to 960 | values
-----
....

I'm shocked to find that the TEC values are different for the same antenna same direction but different subband block, since TEC is a frequency independent quantity. Can someone please clarify what exactly is in those values?

[AHorneffer]

The name of the solution file, L137859_SBgr000-10_uv.dppp.dpppaverage_chunk12_12299B18Ft_0g.merge_selfcal_parmdbs, seems to imply that it contains only the SBs 000 - 009 and only chunk 12 (in time). Is this just a naming convention where the name is just the first SB block and the last chunk?

Yes, the name is an artefact of how the pipeline names output files. The parmDB actually contains all the information for all time- and frequency-steps.

[rvweeren]

What did value did you use for "TEC_block_MHz" in the factor parset ?

[Joshuaalbert]

I left it default of 10.0 which explains the behaviour. Now it seems that small blocks don't give consistent blocks.

[darafferty]

Also, in calculating the convert_* gains which frequencies are used? I have basically just startFreq and endFreq and then numFreq, so I would expect the frequencies should be f_n = startfreq + n * 1/2 (endFreq - startFreq)/numFreq with n = 1...numFreq, though I imagine one could have used startFreq + n * (endFreq - startFreq)/numFreq with n = 0...numFreq-1.

I think the lofar.parmdb library handles this. For the convert_* gains, we just calculate the values at the frequencies returned by the getValuesGrid() method.