There is actually a third option:
3. Allow XA transaction element only on transport specific endpoints that support XA and also allow it on generic endpoints. So you would be able to configure them on "endpoint" and jms:endpoint" but not on "smtp:endpoint" for example.
The approach with 2) would be to have a single element in mule.xsd which is used from each transport schema, there is no need to have transport specific XA transaction elements. See