Non-portable generated code for `size_t`

[jlblancoc]

Problem

// C++ API to be wrapped:
size_t foo();

Generates code replacing size_t with unsigned long. Which is OK... until you try to build the generated sources in another platform (e.g. armhf, etc.) where size_t becomes something else, so the generated pattern code doesn't match anymore the original C++ API and fails to build. A solution might be re-running binder for each architecture, but I would like to avoid it.

Questions

• Within binder, where is the type size_t and such converted into their underlying types?
• What would you think about a new config option to "freeze" certain types such that they are not resolved into their underlying types (if this is even possible...)

If you give me some pointers, I would be glad to attempt to implement it and PR.

[CStewart-Burger]

I just came here for the same thing. I'm not particularly familiar with clang, but I think you'd want to look at https://github.com/RosettaCommons/binder/blob/67a9fe429698d752dc336122995f838bdeb7f92d/source/function.cpp#LL245C5-L245C5.
I've also noticed some other argument types not being parsed as I'd expect. @jlblancoc , have you tested other stdint types?

[jlblancoc]

Thanks for the pointer! I'll try to craft something around that function...

, have you tested other stdint types?

sure, the same happen for examples for uint8_t which gets mapped into unsigned char, but that one is fairly standard and no problematic, but that's not always the case :-(

[lyskov]

@jlblancoc unfortunately in general this is expected behavior and right now i could not offer you any workarounds for solving it. For my main project i have to give on idea of generating binding on one platform and compiling it on the other due to this issue. In short: this issue is due to how LLVM/Clang inner handling of the types is implemented, after function is parsed it types reported in "resolved" form so all information about typedefs is lost.

It has been a while when there was last time i checked on this issue. I will take another look in case newer LLVM versions allows to extract original type info but I think chances of this are slim.

Questions

• Within binder, where is the type size_t and such converted into their underlying types?

-- this conversion is happened inside LLVM/Clang layer so this is not something in our control.

• What would you think about a new config option to "freeze" certain types such that they are not resolved into their underlying types (if this is even possible...)

-- similar ideas was considered in the past. Problem with this approach is that we have no way to determine which type instances of unsigned long should be converted to size_t and which should not.

[lyskov]

@jlblancoc , i just took a fresh look at the code and new API docs and i think there might be a way to get original type information! Let me play with this and i will get back to you.

[jlblancoc]

Thanks, it would be great! In the meanwhile I tried with tricks redefining __SIZE_TYPE__ and alike, but that's a dead end...

[lyskov]

@jlblancoc - it turned out that general solution for this issue might require a bit more research. So for now i found a decent workaround that we can use but it will require listing all std:: types that we confident could be back-converted. Right now i added only std::size_t but it should be possible to add correct handling for any non-template types. So could you please try to pull from recent master and see if this works for you? And if/when you discover similar error try adding appropriate std type into this list: 30e53cc#diff-098881659b3bdd75b9b57555bb70247c118325422929347527db13c1e67bd0faR471 - PR with updated list will be welcome 😬 - as i mentioned this will currently only work for non template types. Thanks,

[jlblancoc]

Sergey, this is definitively awesome, thanks! 👏👏

Sure, I'll try it and expand in a PR after testing...

Cheers

[jlblancoc]

PS: it would add a certain performance loss, but if you prefer it, I could also add an optional set of types loadable from the config file, in addition to the standard std types I will add.

[lyskov]

PS: it would add a certain performance loss, but if you prefer it, I could also add an optional set of types loadable from the config file, in addition to the standard std types I will add.

-- that would be great, PR adding this functionality will be welcome!

Re possible performance hit: - i would not worry about this. We can make sure that standard_names set is only initialized from options once. For instance the most straightforward way to achieve this would be to refactor line 30e53cc#diff-098881659b3bdd75b9b57555bb70247c118325422929347527db13c1e67bd0faR471 to something like this:

static std::set<string> standard_names;
if( standard_names.empty() ) {
  standard_names.insert( {"std::size_t"} );
  <add names from config options here>
}

[jlblancoc]

@lyskov After your fix, all cl.def(..) cases are fixed regarding size_t and friends, that's all good.

But now, the same problem remains in the auto-generated wrappers PyCallBack_*, see for example this where "size_t" has been converted to "unsigned long", even after the latest commits.

I'll try to take a look at binder code to see where those wrappers are generated, but you would probably have it already in your head and provide a faster fix (?).

[jlblancoc]

Maybe here?

https://github.com/RosettaCommons/binder/blob/master/source/class.cpp#L701-L720