Xamarin requires linker hints to preserve APIs accessed via reflection

[Kortdag]

I had an issue using EF in a Xamarin app using a linker process.
According to the community the issue is due to EF using reflection internally which results in the linker removing framework parts that it should not.

Apprarently it can be solved by EF Core adding linker hints...
See
dotnet/macios#3441 (comment)
dotnet/macios#3394

So a suggestion to do so :-)

[divega]

@spouliot A few questions about this:

1. What is the recommended way to figure out what needs to be preserved, just trial an error? (ideally we would like to run our tests on Xamarin.iOS, but not sure that is possible). Or is there some guideline that can simplify the process?
2. What PreserveAttribute do we need to use? Would we need to take a new dependency or can we define our own with the same shape?
3. We had the experience solving similar problems for .NET Native, and it was a very tedious process (we would like to avoid doing again 😄), which resulted in our current rd.xml files. Do you happen to know how the requirements differ between .NET Native and your linker? Assuming the requirements are similar, could the linker consume rd.xml?

[divega]

@spouliot one more thing about rd.xml: the whole platform has gone through the process of figuring out what should be preserved by default, e.g. see System.Linq.Expressions.rd.xml, so unless the requirements for the linker are different, it would be ideal of these could be leveraged.

[cwrea]

@divega On your point 2 above, the [PreserveAttribute] is matched by name and no dependency need be taken on Xamarin libraries.See Xamarin - iOS - Advanced Topics - Linking on iOS. Quoting:

If you do not want to take a dependency on the Xamarin libraries -for example, say that you are building a cross platform portable class library (PCL) - you can still use this attribute.

To do this, you should just declare a PreserveAttribute class, and use it in your code, like this:

public sealed class PreserveAttribute : System.Attribute {
    public bool AllMembers;
    public bool Conditional;
}

It does not really matter in which namespace this is defined, the linker looks this attribute by type name.

[cwrea]

@divega After some brainstorming, here's an idea related to your point 1 and your follow-up comment:

It might not strictly be necessary to run tests on Xamarin.iOS (although that would be ideal) to discover what needs preserving. The types and methods that Xamarin.iOS would need preserved in EF Core are probably the same ones that would need to be preserved for any target using a linker like that used by Xamarin.iOS.

And if I recall correctly, the linker used for Xamarin.iOS is descended from the Mono Linker project. Moreover, the .NET team has built a linker, the .NET IL Linker, based on the Mono Linker. And the .NET IL Linker has listed in its "Caveats" a point that could be a feature to help discover what needs to be preserved in EF Core:

The linker has the following caveats. [...]

• The linker can and will break some apps that use reflection. See Using IL Linker Advanced Features for more information about managing reflection usage.

[divega]

@cwrea thanks for the additional information. This makes standardizing on a single mechanism to specify what reflection metadata needs to be preserved even more compelling.

Still hoping to get an answer from @spuliot on the second and third questions.

[spouliot]

@divega

What is the recommended way to figure out what needs to be preserved, just trial an error? (ideally we would like to run our tests on Xamarin.iOS, but not sure that is possible).

Trial and error won't totally work. E.g.

• That would assume 100% coverage; and even then

• You can reflect Foo sucessfully because the application (or another library or even in your own library) has a reference on Foo. That might not be true for another application and the code will fail;

Running your tests on iOS would require a Mac/Xcode but it's not terribly hard to set up (simulator). However it will only catch what's tested, along with the accompanying code.

Or is there some guideline that can simplify the process?

Audit all of your usage of reflection. Make sure the types (minimally) or the methods (optimally) are preserved by an XML file, [Preserve] attributes or or by code. That's the only safe way to enable the linker on your code.

What PreserveAttribute do we need to use? Would we need to take a new dependency or can we define our own with the same shape?

You can define your own PreserveAttribute as long as it match the one from Xamarin.iOS (and other products using the linker). The namespace can differ too (it's a string-based comparison) only the name and properties really matter.

This was made (long ago) to avoid additional references on user (and 3rd party) code. The linker will remove them from the output (so it has no negative impact of linked application).

You can even add [assembly:LinkerSafe] to your assemblies once you have completed the audit work (that's another attribute you can add inside your own project). That will tell the linker to assume the assembly can be linked by default (just like any other SDK/BCL assemblies we ship with our package).

We had the experience solving similar problems for .NET Native, and it was a very tedious process (we would like to avoid doing again 😄), which resulted in our current rd.xml files. Do you happen to know how the requirements differ between .NET Native and your linker? Assuming the requirements are similar, could the linker consume rd.xml?

I have not used .NET Native so I can't be 100% sure - still it's the same challenges and there's not much latitude in solving them. It looks like the linked XML file could be transformed but it's not the same class libraries (e.g. System.Private.CoreLib) so it won't be a complete/perfect match.

However I know that .NET core is now using the same, base linker code as Xamarin (iOS, Mac and Android) products. Time invested in fixing this should prove useful to all of them.

[cwrea]

@spouliot Re: "Running your tests on iOS would require a Mac/Xcode but it's not terribly hard to set up (simulator)." Confession: I didn't know that builds targeting the simulator will actually use linking when so instructed. I'd just assumed the directive was ignored, but now I'm understanding "Don't Link" is merely a default. Anyway, great to know for diagnosing linking issues specifically ... I'd been doing a lot of that lately and now I'm kicking myself that I didn't know this! :)

But more broadly speaking, and I apologize for veering slightly off topic here (but I'll circle back): I made avoiding the simulator a habit due to another feature-slash-limitation. In the Xamarin iOS Architecture page, this: "If you are running [...] a Xamarin.iOS application on the simulator, the .NET Common Language Runtime (CLR) compiles the MSIL using a Just in Time (JIT) compiler." i.e. Simulator builds don't perform the AOT compilation that's necessary for real devices.

On the one hand, a time saver: You're tweaking UI code and want short edit-build-test cycles — great.

On the other hand, without AOT the simulator's value for testing plumbing code is reduced. False positives arise w.r.t. known limitations as I'm sure you're aware. In my own early quest for a lightweight, cross-platform ORM for use in both .NET web apps and Xamarin.iOS, more than once the simulator gave hope that AOT later snatched away, as the simulator was OK with performing Reflection.Emit().

So, despite having used the simulator for app development with Apple's tooling and languages, I decided to avoid it with Xamarin. I could accept obvious physical limitations (no camera, no GPS, etc.) would preclude simulator use for some feature scenarios, but IMHO even more generally I didn't want to use a tool that could so easily give false hope. "Fool me once..." etc.

Thankfully, Reflection.Emit() isn't in .NET Standard 2.0, and EF Core avoided it early for compatibility with UWP, with iOS benefitting too. But, there remains the possibility that other AOT limitations, and I'm thinking of those w.r.t. generics, could be problematic for .NET Standard libraries having non-trivial plumbing.

So what I'm trying to say is: Perhaps AOT should be an option one could enable in simulator builds? It would make the simulator more useful for .NET Standard library developers aiming for wider compatibility, and for app developers vetting external dependencies they're thinking of taking on.

Or, more generally: imagine tooling or a specialized runtime, both suitable for headless automated testing as well as where linking and AOT are the facts of life. If Xamarin/Mono/Microsoft had something like this available to library developers, whatever their development platform, it certainly wouldn't eliminate the need for proper testing on other operating systems and different CPU architectures. But it would go a long way in raising odds that .NET Standard library code developed with good intentions of being cross-platform would "just work" in the strictest environments.

[spouliot]

@cwrea

as the simulator was OK with performing Reflection.Emit()

Nope, at least not by default. You can enable REPL (optional) but otherwise the mscorlib.dll shipped with XI does not have support for SRE, including none of the types so you'll fail at build time.

other AOT limitations, and I'm thinking of those w.r.t. generics

To be honest the number of limitations are shrunk significally over the years and very few people hit them anymore. You're much more likely to hit sim/device differences - e.g. related to the amount of memory available.

Also the removal of some limitations did have a performance impact, e.g. sharing generic value types. However that's a case you're far more likely to notice if running on devices.

On the one hand, a time saver:

Yeah - it's not, by Apple design, an emulator and it shows in many places beside hardware capabilities (e.g. largely no sandbox in place).

It's very useful because it can be much faster... but for some type of development it does not help (anything the simulator does not support).

So YMMV with the simulator, but it's generally a pretty good tool but it cannot replace at least some testing on devices.

Perhaps AOT should be an option one could enable in simulator builds?

It's been discussed many times. However there's no scenario that really benefits from it.

• It would slow down builds a lot - the main reason why the simulator is useful. It would not be any faster than current device builds;

• It still won't be the same code running - it will be a different backend, x86_64, with its own quirks. It would also hide most performance/memory issues; and

• It won't behave like the device hardware - because Apple does not give us an emulator.

So that would only add one more virtual platform to test and support. One that does not help most developer productivity, does not really help much compatibility and give you an incorrect feeling that things will be fine later. The reality is that you can't avoid some testing on devices.

My suggestion is that if/when you do not care about build times you're much better off testing on the real (device) thing.

If build times are an issue then take an approach similar to ours. We test every commit on the simulator (takes 4-5 hours). But we can't do that on device builds - it's just too slow and the test matrix is too big (it's more than 24h of tests per commit). So the simulator testing act as a filter: it catch most, not all issues, and we get daily updates from devices.