Issue685 - support default values for BCP In on ASE

[mmcnabb-vms]

Summary of fixes on this PR:

• The first three commits fix bugs in bcp in/out , which are actually discussed on Handling of NULL values in FreeBCP native file format with MSSQL #686 . With these fixes in place, data should round-trip correctly using {freebcp|MSbcp|ASEbcp} with character format interchangeably on the same data file. The roundtrip of freebcp -n with SQL Server has had some fixes but there are still outstanding issues regarding nulls in which are covered on 686 and I'll work on at a later date
• Added two new tests and slightly expanded d_bcp test relating to the handling of NULL and Default Values in bcp
• When talking to ASE: implement the feature of ASE bcp that if a column has a Default Value defined in the database, then the column can be omitted from the BCP format file, or have value specified as NULL in the data file where permitted, with the effect that the Default Value for the column will be inserted; even if the column was defined as non-nullable. (MS bcp does not allow omitting columns, and handles default values differently in a way that's already supported by freebcp).
• Allow multiple Hints (that's the stuff in the WITH() clause). I also removed the code that validates the hints, since the database will reject an invalid hint and I don't see what we gain by rejecting the hint on client side; it seems more forwards-compatible to let the database handle this.
• Implement the -k flag as supported by MSBCP (ASE bcp calls it --ignoreDefaults) which means that even if a column has a default value defined, BCP inserting a NULL will actually insert NULL and not the Default Value.

Most of the content is covered by the new test cases, because freebcp is a fairly thin wrapper around DBlib bcp API. However I also have made a local test script that does a few "front door" tests calling freebcp directly to test the null/default handling, and did ad-hoc testing of -k.

[mmcnabb-vms]

Re. the old Ucko PRs - A large part of them was focused on having freebcp in reject malformed data . There are quite a few cases in freebcp in where it will just proceed and go off the rails, including reading the wrong number of bytes from the input file and getting out of sync, in response to various cases of malformed data. Ucko had to try and fix that for security reasons. That would be a separate topic to this PR though. I didn't see anywhere in those PRs that addressed ASE defaults.

[mmcnabb-vms]

Force pushed to fix linux build errors & missing Signoff message .

I noticed while fixing failure in t0016 that t0016 is actually a BCP test framework with 16 sub-cases of BCP tests. I should be able to replace my new test d_bcp_defaultdate with another case for t0016 , will do that now and force-push if you haven't started reviewing yet.

Am also working on a test that will read the same sources as t0016 but execute by calling freebcp (or even MS/ASE native bcp for reference) instead of using DBlib api.

(Edit: Completed that t0016 expansion now)

[mmcnabb-vms]

I'm not fully happy with the commit "freebcp: also report if rows failed to copy": it only reports rows that parsed successfully in the hostfile, but were unable to be packed for upload to the server (or rejected by the server). I think it would be even better to also report rows rejected in the hostfile, and possibly warn more prominently and/or exit with a failure status code if the number of rows copied does not match the number of rows in the hostfile. It seems to me that any row in the dataset failing to upload is something the user will want to be sure to be alerted of.

[mmcnabb-vms]

Known outstanding issues:

• bcp upload of NULL Blob corrupts ASE table #696 (EDIT 10/Feb/2026: Since been fixed in master)
• freebcp in VMS runs a lot more slowly than in Windows, to the same remote server (EDIT 10/Feb/2026: this is a problem with overhead in VMS fwrite)
• I have built a rig to execute the test cases in src/dblib/unittests/t0016_* via freebcp.exe instead of via the DBlib API. The tests that use unicode characters fail when executed from Windows; just investigating if that's just a windows terminal setting or a larger problem. (EDIT 10/Feb/2026: Had to create a DSN with "client charset = UTF-8" for these tests; note that we cannot use that setting for the rest of the test suite as that makes some other tests fail).

Otherwise, ready for review. There are two commits near the end regarding datarows-locking that fix bugs in my previously-merged code for datarows-lock support (it didn't support temp-tables, and had a buffer overflow)

[mmcnabb-vms]

@freddy77 I see in your 1.5.11 update that fixed the null BLOB problem there is some overlap with this PR , what are your plans here?

Also I have ended up with this PR containing both the default values change, and a bunch of other bcp improvements that strictly speaking, aren't to do with default values. I could try to separate the other improvements to a new PR perhaps?

[freddy77]

It starts being time to prepare for a new release, there are quite some new features and improvements, especially in BCP code.
I would like to integrate most of the BCP fixes/changes before the release.

Yes, a rebase on master would help.

[mmcnabb-vms]

It starts being time to prepare for a new release, there are quite some new features and improvements, especially in BCP code. I would like to integrate most of the BCP fixes/changes before the release.

Yes, a rebase on master would help.

OK. I see now Ucko PR #558 where he had already done the Defaults change, managed to overlook that when checking his other PRs. The other changes on #558 look at first sight as if they could come in to master without too much trouble

[mmcnabb-vms]

Will close this and reissue a new request on master next week