Implement linker list

[rikkimax]

Requires DIP, changelog and test.

I can't be bothered to do it atm.

Uses sections for within binary appending of symbols with CTFE appending.

I want to emphasise that, if a user were to attempt to do this, they will very likely fail.
Being a compiler feature allows us to ensure it is correct codegen and brings us in line with the capabilities of application VM languages.

[dlang-bot]

Thanks for your pull request and interest in making D better, @rikkimax! We are looking forward to reviewing it, and you should be hearing from a maintainer soon.
Please verify that your PR follows this checklist:

• My PR is fully covered with tests (you can see the coverage diff by visiting the details link of the codecov check)
• My PR is as minimal as possible (smaller, focused PRs are easier to review than big ones)
• I have provided a detailed rationale explaining my changes
• New or modified functions have Ddoc comments (with Params: and Returns:)

Please see CONTRIBUTING.md for more information.

---

If you have addressed all reviews or aren't sure how to proceed, don't hesitate to ping us with a simple comment.

Bugzilla references

Your PR doesn't reference any Bugzilla issue.

If your PR contains non-trivial changes, please reference a Bugzilla issue or create a manual changelog.

Testing this PR locally

If you don't have a local development environment setup, you can use Digger to test this PR:

dub run digger -- build "master + dmd#22824"

[rikkimax]

Looks like I've got a null deref or something.

Oh and I also need to do c++ headers fix.

[ibuclaw]

This looks like something that could be handled with no language support if there was a drtbeg.o/drtend.o object that the compiler passes to the linker to make section boundaries for declared @sections.

[rikkimax]

This looks like something that could be handled with no language support if there was a drtbeg.o/drtend.o object that the compiler passes to the linker to make section boundaries for declared @sections.

No, the sections themselves are user-provided.

You don't know what exists with multi-step compilation which is the norm for D.

[rikkimax]

Also start + end symbols only need to exist for Windows, the rest are all special ones that don't require actual global variables associated with them. They can be emitted wherever the iteration is. So they are a non-issue.

Given the constraints on mutability, types, @used, let alone CTFE appending, you wouldn't even be solving 1/3'rd of the problems that the compiler support does, even if they were to work (which they won't due to build steps).

[rainers]

@section should be good enough to implement this in a library. This works as needed under Windows with LDC:

import core.stdc.stdio;
import core.attribute;

template getSectionStart(string name)
{
	@section(name ~ "$A") __gshared int specialStart = 1; // moved into .data by dmd
	void* getSectionStart()
	{
		return &specialStart;
	}
}

void* getSectionEnd(string name)()
{
	@section(name ~ "$Z") __gshared int specialEnd; // moved into .bss by dmd
	return &specialEnd;
}

@section(".special$B") __gshared int specialGlobal;

void* foo()
{
	@section(".special$B") __gshared int specialFoo;
	return &specialFoo;
}

void main()
{
	@section(".special$B") __gshared int specialMain;
	
	printf("specialStart @ %p\n", getSectionStart!".special");
	printf("specialGlobal @ %p\n", &specialGlobal);
	printf("specialFoo @ %p\n", foo());
	printf("specialMain @ %p\n", &specialMain);
	printf("specialEnd @ %p\n", getSectionEnd!".special");
}

It does not work with dmd from master, though. AFAICT @section is ignored inside templates ATM.

[rikkimax]

@section should be good enough to implement this in a library. This works as needed under Windows with LDC:
snip
It does not work with dmd from master, though. AFAICT @section is ignored inside templates ATM.

I implemented the section attribute in dmd.

Yes it works in templates, however this does not do what is being wanted here, nor does it give the reliability.
There is simply too many conditions for it to work correctly, that I can hide in compiler but not in user code.

I also want to emphasise, I want to be able to append to the list using CTFE, and have it end up in the appropriete root module.

[rainers]

Yes it works in templates

Not in this example. The section is created, but the variables are still in data/bss:

SECTION HEADER #13
 .data$B name
       0 physical address
       0 virtual address
       4 size of raw data
    1024 file pointer to raw data (00001024 to 00001027)
       0 file pointer to relocation table
       0 file pointer to line numbers
       0 number of relocations
       0 number of line numbers
C0501040 flags
         Initialized Data
         COMDAT; sym= _D11testsection__T15getSectionStartVAyaa8_2e7370656369616cZ12specialStarti
         16 byte align
         Read Write
[...]
04A 00000000 SECT13 notype  External | _D11testsection__T15getSectionStartVAyaa8_2e7370656369616cZ12specialStarti
[...]
   Summary
           0 .special$A
           C .special$B
           0 .special$Z

append to the list using CTFE, and have it end up in the appropriete root module.

What do you mean by "CTFE appending"? Can it do more than @section(calcSection()) auto var = ctfeFunc();.

Can you add a test case (or provide a use case) that does something that is not possible yet.

[rikkimax]

Yes it works in templates

Not in this example. The section is created, but the variables are still in data/bss:

SECTION HEADER #13
 .data$B name
       0 physical address
       0 virtual address
       4 size of raw data
    1024 file pointer to raw data (00001024 to 00001027)
       0 file pointer to relocation table
       0 file pointer to line numbers
       0 number of relocations
       0 number of line numbers
C0501040 flags
         Initialized Data
         COMDAT; sym= _D11testsection__T15getSectionStartVAyaa8_2e7370656369616cZ12specialStarti
         16 byte align
         Read Write
[...]
04A 00000000 SECT13 notype  External | _D11testsection__T15getSectionStartVAyaa8_2e7370656369616cZ12specialStarti
[...]
   Summary
           0 .special$A
           C .special$B
           0 .special$Z

It should be working:

dmd/compiler/test/runnable/sectiondefs.d

Line 46 in ca244ef

__gshared

The code isn't checking for things like templates.

And yes I was doing objdump to verify symbols going into sections, and they are iterating correctly.

append to the list using CTFE, and have it end up in the appropriete root module.

What do you mean by "CTFE appending"? Can it do more than @section(calcSection()) auto var = ctfeFunc();.

Yes. You are not limited to one per declaration.

It works in loops as a statement.

But most importantly, anyone can use this and get it right. You don't need to know any platform specific details (which there are a bunch). There are fun stuff like only pointers can go into a linker list, I can hide all these awful details.

Can you add a test case (or provide a use case) that does something that is not possible yet.

Note: no tests are implemented yet, for this PR its not going to be fun to actually get it into the test runner.

I have test code locally running on Windows.

import second;
import third;

@MyUDA(2)
@MyUDA(9)
@MyUDA(1)
void main() {
    import core.stdc.stdio;

    foreach(ref v; myInts) {
        printf("I got: %d\n", v);
        v++;
    }

    foreach(v; myInts) {
        printf("I mutated: %d\n", v);
    }

    foreach(v; myObjects) {
        printf("got object %p\n", v);
    }

    foreach(v; myIntefaces) {
        printf("got interface %p\n", v);
    }
}

static assert(() { myInts ~= 6; return true; }());
static assert(() { myObjects ~= new Object; return true; }());
static assert(() { myIntefaces ~= new CI; return true; }());

module second;
import third;

struct MyUDA {
    this(int val) {
        if (__ctfe)
            myInts ~= val;
    }
}

module third;
import core.attribute;

__linkerlist!int myInts;
__linkerlist!Object myObjects;
__linkerlist!I myIntefaces;

interface I {}

class CI : I {}

I've had 12 years to think this feature over, this is the feature I needed when I was working on a web service framework.
I have yet to be able to reproduce the capabilities of the play framework that I got to enjoy back in 2010 in D because we're missing features like this to finish off the UDA design.

[rainers]

So I guess your main use case is generating runtime data from UDAs. Modifying global state by CTFE is unprecedented in D, though, and likely to cause trouble (evaluation order, speculative compilation, etc). Good luck getting this approved.

With a few things fixed with section emission (segment overwritten by .bss/.data in COMDATs, section name length > 8 working differently), this adds data for each UDA:

import core.stdc.stdio;
import core.attribute;

template sectionStart(string name)
{
	@section(name ~ "$A") __gshared int sectionStart = 0; // should be COMDAT
}

template sectionEnd(string name)
{
	@section(name ~ "$Z") __gshared int sectionEnd = 0; // should be COMDAT
}

@section(".spec$B") __gshared int specialGlobal = 11;

struct MyUda(int n, string mod = __MODULE__, int line = __LINE__)
{
	@section(".spec$B") __gshared static int value = n; // a COMDAT with LDC, but not with DMD
}

@MyUda!2()
@MyUda!9()
@MyUda!1()
void main()
{
	@section(".spec$B") __gshared int specialMain = 12;
	
	auto start = &sectionStart!".spec";
	auto end = &sectionEnd!".spec";
	for (auto p = start; p <= end; p++)
		printf("%p = %x\n", p, *p);
}

Output:

00007FF611735000 = 0
00007FF611735004 = b
00007FF611735008 = c
00007FF61173500C = 2
00007FF611735010 = 9
00007FF611735014 = 1
00007FF611735018 = 0

It would be good to allow specifying whether the section is a COMDAT or not, both have their application. LDC and DMD seem to disagree when inside a template.

[rikkimax]

So I guess your main use case is generating runtime data from UDAs. Modifying global state by CTFE is unprecedented in D, though, and likely to cause trouble (evaluation order, speculative compilation, etc). Good luck getting this approved.

Yes it is runtime data from UDA's. It basically gives us the ability to create our own reflection mechanisms.
Except with the trigger points + what to do with the CT transformation being defined. Both of these are not currently supported, UDA's are really only a 1/3rd complete without this solved.

As for CTFE, this isn't actually global state, due to it not being readable at CT.
There is no way to go linker list declaration -> entry, that information isn't stored.

Evaluation order shouldn't matter, there is no guarantee on ordering, or even that the entries are in contiguous memory. Thank you Microsoft.

Speculative compilation again, shouldn't matter too much, the point is to trigger registration of information about symbols that exist. If a symbol doesn't exist, we don't care about it.
I've also slapped on to the linker list entries no-gc flag for the linker, so that shouldn't cause us problems either.
Main issue I foresee is static libraries omitting object files.
But that's an existing issue of shared libraries.

With a few things fixed with section emission (segment overwritten by .bss/.data in COMDATs, section name length > 8 working differently), this adds data for each UDA:

I need to go back and fix section name length > 8, right now it just creates new sections each time, which isn't ideal.
That limitation <=8 had to be yoinked out, and while the existing code works... it simply creates more sections than required.

import core.stdc.stdio;
import core.attribute;

template sectionStart(string name)
{
	@section(name ~ "$A") __gshared int sectionStart = 0; // should be COMDAT
}

template sectionEnd(string name)
{
	@section(name ~ "$Z") __gshared int sectionEnd = 0; // should be COMDAT
}

@section(".spec$B") __gshared int specialGlobal = 11;

struct MyUda(int n, string mod = __MODULE__, int line = __LINE__)
{
	@section(".spec$B") __gshared static int value = n; // a COMDAT with LDC, but not with DMD
}

@MyUda!2()
@MyUda!9()
@MyUda!1()
void main()
{
	@section(".spec$B") __gshared int specialMain = 12;
	
	auto start = &sectionStart!".spec";
	auto end = &sectionEnd!".spec";
	for (auto p = start; p <= end; p++)
		printf("%p = %x\n", p, *p);
}

Output:

00007FF611735000 = 0
00007FF611735004 = b
00007FF611735008 = c
00007FF61173500C = 2
00007FF611735010 = 9
00007FF611735014 = 1
00007FF611735018 = 0

It would be good to allow specifying whether the section is a COMDAT or not, both have their application. LDC and DMD seem to disagree when inside a template.

I've disabled COMDAT's for sections, both attribute and linker list.
Both GCC and LLVM have their own way of configuring sections handling, and dmd will probably not be an exception to this rule if we need it.
I want to see a bug report on that before considering touching it again.
Most likely the current approach is "good enough" for why somebody might need it for dmd.

P.S.

Enjoy: https://devblogs.microsoft.com/oldnewthing/20190114-00/?p=100695 https://devblogs.microsoft.com/oldnewthing/20181108-00/?p=100165