Convert input handling based on CCSID scenarios

[worksofliam]

Enhance input conversion logic to account for CCSID in various contexts, ensuring proper translation for system commands and SQL statements. Adjustments made to improve path handling and user feedback during checks.

Fixes #2519

How to test

1. List members/objects in the Object Browser with variant chats
2. Open source members up
3. Compile source members
4. Run the test cases

[github-actions]

👋 A new build is available for this PR based on 79e49f1.

• Download here.
• Read more about how to test

[sebjulliand]

It used to be inAmerican, so I think this should be forSystem here.

Suggested change

	Tools.qualifyPath(fromSystem(file.library), fromSystem(file.name), fromSystem(member), undefined, localVariants)
	Tools.qualifyPath(fromSystem(file.library), forSystem(file.name), forSystem(member), undefined, localVariants)

[sebjulliand]

The logic behind the enabled parameter is unclear or would need to be redefined. Looking at all the references in the code, it's always this class' qsysPosixPathsRequireTranslation field that is used.

Here is a suggestion: make that second parameter optional, and when it's true, consider qsysPosixPathsRequireTranslation value. Then in the places where the method is called with the second parameter, pass ifNeeded as true:

Suggested change

	sysNameInAmerican(string: string, enabled: boolean = true) {
	if (enabled) {
	const fromChars = this.variantChars.local;
	const toChars = this.variantChars.american;
	let result = string;
	let result = string;
	for (let i = 0; i < fromChars.length; i++) {
	result = result.replace(new RegExp(`[${fromChars[i]}]`, `g`), toChars[i]);
	};
	for (let i = 0; i < fromChars.length; i++) {
	result = result.replace(new RegExp(`[${fromChars[i]}]`, `g`), toChars[i]);
	};
	return result;
	}
	return result
	return string;
	}
	sysNameInAmerican(string: string, ifNeeded?: boolean) {
	if (!ifNeeded || this.qsysPosixPathsRequireTranslation) {
	const fromChars = this.variantChars.local;
	const toChars = this.variantChars.american;
	let result = string;
	for (let i = 0; i < fromChars.length; i++) {
	result = result.replace(new RegExp(`[${fromChars[i]}]`, `g`), toChars[i]);
	};
	return result;
	}
	return string;
	}

[sebjulliand]

@worksofliam I made an initial code review. I'll see how it goes at runtime.
I left a few comments for some changes. Thanks!

[sebjulliand]

Tested with CCSID 297: listing files and members from a single library with variants work.

Opening a member whose path contains variants doesn't work. When source dates are enabled, the SQL alias is created with American variants:

CREATE OR REPLACE ALIAS ILEDITOR.TEMP_E3D5CD600772FF279A8AF823A8DBB3901555096A for "@MAZING"."H@RRIBLE"("H@L@#@$")

It used to be created with the locale's variants:

CREATE OR REPLACE ALIAS ILEDITOR.TEMP_6790C94968740AB09E5F395BAFD5AC15037BB33A for "àMAZING"."HàRRIBLE"("HàLP£à$")

Listing libraries with a variant doesn't work:

it used to list four libraries:

The reason is the SQL query used to use the variant chars:

select    OBJNAME          as NAME,   OBJTYPE          as TYPE,   OBJATTRIBUTE     as ATTRIBUTE,   OBJTEXT          as TEXT,   0                as IS_SOURCE,   IASP_NUMBER      as IASP_NUMBER,   OBJSIZE          as SIZE,   extract(epoch from (OBJCREATED))*1000       as CREATED,   extract(epoch from (CHANGE_TIMESTAMP))*1000 as CHANGED,   OBJOWNER         as OWNER,   OBJDEFINER       as CREATED_BY from table(QSYS2.OBJECT_STATISTICS(OBJECT_SCHEMA => 'QSYS', OBJTYPELIST => '*LIB', OBJECT_NAME => 'à*'))

With the PR, it doesn't:

select    OBJNAME          as NAME,   OBJTYPE          as TYPE,   OBJATTRIBUTE     as ATTRIBUTE,   OBJTEXT          as TEXT,   0                as IS_SOURCE,   IASP_NUMBER      as IASP_NUMBER,   OBJSIZE          as SIZE,   extract(epoch from (OBJCREATED))*1000       as CREATED,   extract(epoch from (CHANGE_TIMESTAMP))*1000 as CHANGED,   OBJOWNER         as OWNER,   OBJDEFINER       as CREATED_BY from table(QSYS2.OBJECT_STATISTICS(OBJECT_SCHEMA => 'QSYS', OBJTYPELIST => '*LIB', OBJECT_NAME => '@*')

Three encoding tests fail with this config:

 FAIL  tests/suites/encoding.test.ts > Encoding tests > Listing objects with variants
AssertionError: expected undefined to be truthy

- Expected: 
true

+ Received: 
undefined

 ❯ tests/suites/encoding.test.ts:223:33
    221|       if (!skipLibrary) {
    222|         const [expectedLibrary] = await content.getLibraries({ library });
    223|         expect(expectedLibrary).toBeTruthy();
       |                                 ^
    224|         expect(library).toBe(expectedLibrary.name);
    225| 

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[7/9]⎯

 FAIL  tests/suites/encoding.test.ts > Encoding tests > Library list supports dollar sign variant
Error: Failed uploading member: Error: Erreurs trouvées dans la commande. (38501)
 ❯ IBMiContent.uploadMemberContent IBMiContent.ts:291:15
    289|         return true;
    290|       } else {
    291|         throw new Error(`Failed uploading member: ${copyResult.stderr}`);
       |               ^
    292|       }
    293|     } catch (error) {
 ❯ tests/suites/encoding.test.ts:290:7

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[8/9]⎯

 FAIL  tests/suites/encoding.test.ts > Encoding tests > Variant character in source names and commands
Error: Failed uploading member: Error: Erreurs trouvées dans la commande. (38501)
 ❯ IBMiContent.uploadMemberContent IBMiContent.ts:291:15
    289|         return true;
    290|       } else {
    291|         throw new Error(`Failed uploading member: ${copyResult.stderr}`);
       |               ^
    292|       }
    293|     } catch (error) {