The Philistines can’t wait to be rid of the ark, which, instead of making them successful, oppresses them. Imagine being the people of Beth-shemesh out working in their fields and seeing the ark being returned. The excitement is short-lived as God punishes them for not treating the ark with respect.
Although symbols connected with God cannot be used as good luck charms, God does demand our respect and obedience.