Why would it seem appropriate when you didnt do anything wrong? Think about this for a moment.... If you really were penalised, then a dishonest golfer would never get beaten, he could just accidentally pick up the other guys marker whenever it suited him. So it would not really make any sense for you to be penalised at all.
The thing you really should have done is play out the hole as your group suggested and asked for a ruling later.
FYI, even if you accidentally pick up your own marker its only a 1 stroke penalty.
and last important point, this situation is in the rule book. No real need for wondering