Accepted types of summary stat inputs

[edg1983]

We all agree to accept both compressed and uncompressed text files as summary stats input. However, we need to decide how flexible we want to be here in terms of formats / delimiters.

Possible options in my mind are

• enforce a specific format, which poses some limitations on the user but makes our life easier
• Allow for different delimiters, but ask the user to configure them explicitly in the input file table (adding a new column for delimiter). This is a compromise, more work for the user but avoids uncertainty
• let the reading function guess. This is generally fine unless we think the format of summary stats can be largely variable, and thus, the automatic guess may result in incorrect parsing. In my mind, they are pretty simple files. What do you think based on your experience @ariannalandini @Sodbo @bruno-ariano ?

Based on the result of this discussion, we can adapt all the scripts accordingly.

If we go for the third option, it would be good to update the test datasets so that we have a mix of delimiters.

[bruno-ariano]

I am open to the first and second option but I would not use the third one. I had few issues in the past parsing files in R where the files had mixed separators tab and spaces together. I would also suggest for the second option to not give the possibility of using spaces as separators.

[edg1983]

@ariannalandini ... I kinda agree with Bruno here. We may want to enforce tab-delimited input files for summary stats (plain text of gzipped), since this is by far the most popular format in my mind.

What do you think? How often did you have to process summary stats that were space or comma-separated?

[ariannalandini]

In my experience, GWAS summary stats are typically tab-separated. @Sodbo disagrees, as in his experience comma-separated files are just as common. While this isn’t a critical issue - since users can still use the pipeline if we clearly specify the expected format - can we enforce tab-seprated files for the 1.0 release, and leave this issue open for the next release, as user's quality of life improvement?

[edg1983]

We can do as you think it's best.

The decision affects how I will implement the file reading function.
Since you tell me that comma-separated is possible and we may want to include this in the future, I suggest we use the generic read_delim (I'm using the more strict read_tsv now). Then, we can enforce sep=" t" for now.

In this way, if we want to support other separators in the future, we can remove the sep option and let the function guess or provide the separator character explicitly through a parameter or some other logic.

I see two minor drawbacks if do not set sep:

1. Slightly slower reading since the guessing procedure takes a moment (impact is minimal)
2. Less control over the file structure and, thus, potential parsing errors. This is an issue only if we expect messy files sometimes. In my mind it is not the case with summary stats, can you confirm?

Do you think this is a good solution?

[ariannalandini]

Sounds good to me, I don't expect messy summary statistics (not on purpose at least). @bruno-ariano will also the input validation py code be able to guess the delimiter? It's fine if only tab and comma are given as possible options.

[bruno-ariano]

From my side I've found that I can also accept multiple format but I have to define these in the reading function as regex format. So I can add tab and comma for now and later we can add other ones if there is the necessity