Potential Energy
Potential energy is stored in the configuration of an interacting system. Its zero is a reference choice; changes in potential energy are what connect physical states.
If the zero level is arbitrary, what part of gravitational potential energy is physically meaningful?
Changing the reference shifts every potential-energy value by the same amount. The difference between two states stays unchanged.
Gravitational potential energy belongs to the Earth-object system. Elastic potential energy belongs to the spring-object system. The configuration of interacting parts is what matters.
Move the zero reference. Both absolute values shift, but stays the same because the same reference offset appears in both states and cancels.
Potential energy is broader than height. Springs, electric charges, chemical bonds, and many other interactions can store energy in configuration.