I have tried reproducing what you've been doing and my observation is that as soon as you modify the Attribute Templates of an Element Template, there is impact to the Attributes of derived Elements. With other words if you remove an Attribute Template you cause the corresponding Attribute to be removed from derived Elements. When you add an Attribute Template, the corresponding Attribute will be added to Elements derived from your Element Template and being initiated with the value specified in the Template.
The only way to implement the logic you have in mind is to see what Attributes exist with the current Element Template against what is specified in the XML export. You would keep those Attribute Templates untouched which exist in both, remove those which exist only in the current Template and add those which do not exist in the Template but in the XML file. However, there is always impact to the Attributes of derived Elements.
I discussed this with a couple of colleagues today as well, and we came to the same conclusion.
It would mean that I need to dig deeper into the XML, by using more specialized XML comparers instead of what I originally planned to use - a simple string compare. But I feel this is a must-have if you're dealing with a production-database with lots of derived elements. (If this was purely a template-store database, we would probably be alright taking the easy way.)