1558 csv data reader by alexfurmenkov · Pull Request #1640 · cdisc-org/cdisc-rules-engine

alexfurmenkov · 2026-02-26T11:17:15Z

Added support for data in .csv format.
Implemented params reading from .env files - based on .env in data folder.

SFJohnson24

This looks good so far-- can you see messages on slack with me and Gerry re: the .env and taking that into account as well with csv runs. https://github.com/cdisc-org/cdisc-rules-engine/issues/1559 My parsing script matches your reader logic--there is a description of the .env structure there.
It would want to get the standard/version/substandard/CT/define xml path from the .env

…-dp, -d] parameters to run validation.

RamilCDISC · 2026-03-05T20:32:59Z

+        raise NotImplementedError
+
+    def from_file(self, file_path):
+        with open(file_path, "r", encoding=self.encoding) as fp:


We will need encoding + general error handling wrapping similar to json reader here.

RamilCDISC · 2026-03-05T20:34:53Z

+            first_record = {}
+
+        with open(self.file_path, encoding=self.encoding) as f:
+            dataset_length = sum(1 for _ in f) - 1  # subtract header


This can result in dataset_length = -1 for empty csv files. For those cases I guess it should be zero. So we should clamp it to zero.

RamilCDISC · 2026-03-09T23:22:33Z

    description = "JSON data is malformed."


+class InvalidCSVFormat(EngineError):


I think better naming for this would be InvalidCSVFile instead of format because this error is for when csv is malformed or encoding failed which makes it invalid csv file. What do you think?

SFJohnson24 · 2026-04-02T16:04:03Z

    type=click.Path(exists=True, readable=True, resolve_path=True),
    help="Path to directory containing local rules.",
    multiple=True,
+    envvar="LOCAL_RULES",


no LOCAL_RULES

SFJohnson24 · 2026-04-02T16:05:01Z

    required=False,
    help="Rule ID to get rule for.",
    multiple=True,
+    envvar="RULEID",


SFJohnson24

noticed a lot of env params that dont exist:

PRODUCT=TIG
VERSION=1-0
SUBSTANDARD=SDTM
CT=sdtmct2020-12-18
DEFINE_XML=define_negative1.xml

above is a complete .env file with all possible params from the cdisc-open-rules repository. You can hop on the rules branch and see what it looks like if you want

RamilCDISC · 2026-04-02T19:35:11Z

    required=True,
    default=None,
    help="Standard version to validate against",
+    envvar="VERSION",


All else looks fine to me. Could you please adda test where we check the .env interoperability. You can cover all the flags in few tests.

…ut to match other readers logic

# Conflicts: # README.md

RamilCDISC

I ran the following command in the cli:
python core.py validate -s sdtmig -v 3.4 -dp tests/resources/CoreIssue1558/datasets/dm.csv -dep tests/resources/CoreIssue1558/test.env

The obtained repor tis attached. There are errors for rule CORE-000658 CORE-000711 CORE-000713 CORE-000714. The issue details sheet shows the error in the Value(s) column.

CORE-Report-2026-04-06T15-00-16.xlsx

RamilCDISC · 2026-04-07T21:58:31Z

@SFJohnson24 Could you please confirm if you agree with the changes made on your request. I have done manual validation and as well as code review. All seems good to me. Once you give a heads up i will add final comment and approve.

…disc-rules-engine into 1558-csv-data-reader

SFJohnson24 · 2026-04-08T17:40:25Z

 MAX_REPORT_ROWS = 10  # integer for maximum number of issues per excel sheet (plus headers) in result report.  Defaults to 10000.
 MAX_ERRORS_PER_RULE = (10, True)  # Tuple for maximum number of errors to report per rule during a validation run. Also has a per dataset flag described as second bool value in readme. example value 
+DEFINE_XML
+DATA_DIR


what is DATA_DIR? Is this an artifact?

should this example file also have the other variables the .env can handle?

SFJohnson24 · 2026-04-08T17:43:09Z

        "CDISC TIG Use Case for scoping a TIG Validation."
        "Any of INDH, PROD, NONCLIN, or ANALYSIS."
    ),
 )


I tested this on TIG and it we will need envvar="USE_CASE" for this standard. I updated the open rules to reflect USE_CASE as an .env variable

SFJohnson24

this looks good--just 1 small tweak that came up in my TIG testing. Looks good for other standards though

SFJohnson24 · 2026-04-08T17:50:48Z

i pushed the 2 lines up just to expedite this getting merged--looks good @alexfurmenkov

alexfurmenkov added 5 commits February 25, 2026 15:27

#1558 WIP: csv readers

9f5273b

#1558 csv metadata reader and tables filtering

c791d7c

#1558 example files

1490237

#1558 moved csv metadata reader

83d2704

Merge branch 'main' into 1558-csv-data-reader

4cbb541

alexfurmenkov temporarily deployed to DEV February 26, 2026 11:17 — with GitHub Actions Inactive

alexfurmenkov added 4 commits February 27, 2026 13:58

#1558 unit tests for dataset filtering

8098eb4

#1558 unit tests for dataset readers

514b147

#1558 regression and changes in metadata reader logic to preserve data

155a236

Merge branch 'main' into 1558-csv-data-reader

b3e8974

alexfurmenkov marked this pull request as ready for review March 2, 2026 08:56

alexfurmenkov requested review from RamilCDISC, SFJohnson24 and gerrycampion March 2, 2026 08:56

SFJohnson24 requested changes Mar 4, 2026

View reviewed changes

alexfurmenkov linked an issue Mar 5, 2026 that may be closed by this pull request

Support CSV datasets in engine #1558

Closed

#1558 added envvar options for [-r, -er, -lr, -ss, -v, -s, -l, -dxp, …

c351222

…-dp, -d] parameters to run validation.

alexfurmenkov temporarily deployed to DEV March 5, 2026 14:15 — with GitHub Actions Inactive

#1558 added envvar option for -ct and -dv parameters.

21b0d3e

alexfurmenkov temporarily deployed to DEV March 5, 2026 14:38 — with GitHub Actions Inactive

Merge branch 'main' into 1558-csv-data-reader

695e1da

RamilCDISC temporarily deployed to DEV March 5, 2026 20:15 — with GitHub Actions Inactive

Merge branch 'main' into 1558-csv-data-reader

0fc3bb1

RamilCDISC temporarily deployed to DEV March 5, 2026 20:24 — with GitHub Actions Inactive

RamilCDISC requested changes Mar 5, 2026

View reviewed changes

#1558 error handling while reading CSV

d38fe58

alexfurmenkov temporarily deployed to DEV March 6, 2026 13:13 — with GitHub Actions Inactive

alexfurmenkov requested review from RamilCDISC and SFJohnson24 March 9, 2026 08:02

RamilCDISC requested changes Mar 9, 2026

View reviewed changes

SFJohnson24 reviewed Apr 2, 2026

View reviewed changes

SFJohnson24 requested changes Apr 2, 2026

View reviewed changes

#1558 removed unnecessary envvars from validation parameters.

176c703

alexfurmenkov temporarily deployed to DEV April 2, 2026 16:52 — with GitHub Actions Inactive

RamilCDISC requested changes Apr 2, 2026

View reviewed changes

#1558 added test to load environment values. Changed from_file() outp…

f421df9

…ut to match other readers logic

alexfurmenkov temporarily deployed to DEV April 3, 2026 14:41 — with GitHub Actions Inactive

Merge branch 'main' into 1558-csv-data-reader

4696ee8

# Conflicts: # README.md

alexfurmenkov temporarily deployed to DEV April 6, 2026 12:02 — with GitHub Actions Inactive

alexfurmenkov requested review from RamilCDISC and SFJohnson24 April 6, 2026 12:24

RamilCDISC requested changes Apr 6, 2026

View reviewed changes

alexfurmenkov requested a review from RamilCDISC April 7, 2026 17:11

changed Nan values to None in dataset

12c7b15

alexfurmenkov temporarily deployed to DEV April 7, 2026 18:17 — with GitHub Actions Inactive

fix unique tables.csv paths

17e2698

alexfurmenkov temporarily deployed to DEV April 8, 2026 08:24 — with GitHub Actions Inactive

Merge branch '1558-csv-data-reader' of https://github.com/cdisc-org/c…

6274e16

…disc-rules-engine into 1558-csv-data-reader

SFJohnson24 reviewed Apr 8, 2026

View reviewed changes

SFJohnson24 requested changes Apr 8, 2026

View reviewed changes

use case

795e14a

SFJohnson24 temporarily deployed to DEV April 8, 2026 17:49 — with GitHub Actions Inactive

SFJohnson24 self-requested a review April 8, 2026 17:53

SFJohnson24 approved these changes Apr 8, 2026

View reviewed changes

SFJohnson24 merged commit eea0443 into main Apr 8, 2026
12 checks passed

SFJohnson24 deleted the 1558-csv-data-reader branch April 8, 2026 17:54

		description = "JSON data is malformed."


		class InvalidCSVFormat(EngineError):

Conversation

alexfurmenkov commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SFJohnson24 left a comment

Choose a reason for hiding this comment

Uh oh!

RamilCDISC Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

RamilCDISC Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

RamilCDISC Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

SFJohnson24 Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

SFJohnson24 Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

SFJohnson24 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RamilCDISC Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

RamilCDISC left a comment

Choose a reason for hiding this comment

Uh oh!

RamilCDISC commented Apr 7, 2026

Uh oh!

SFJohnson24 Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

SFJohnson24 Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SFJohnson24 Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SFJohnson24 left a comment

Choose a reason for hiding this comment

Uh oh!

SFJohnson24 commented Apr 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alexfurmenkov commented Feb 26, 2026 •

edited

Loading

SFJohnson24 left a comment •

edited

Loading

SFJohnson24 Apr 8, 2026 •

edited

Loading

SFJohnson24 Apr 8, 2026 •

edited

Loading