As you note, the lower voltage coil will operate at higher magnetizing current for the core to be operated at the same ampere-turns.
In addition to this factor, a transformer designed as a step down, but operated as a step up will have higher inrush current that a transformer of the same nominal rating and same over-all build quality, but designed from the get-go as a step up.
My understanding is that the primary coil is generally placed outside of the secondary coil, so that there is more air path leakage flux seen by the primary coil, and this air path leakage flux reduces the inrush current. When you operate a transformer in 'reverse', then what should be the secondary coil is being used as the primary, and you don't have the benefit of the leakage flux reducing inrush.
-Jon