In RCO2H, which oxygen (the carbonyl oxygen or the protonated oxygen) is more nucleophilic and why?

First, RCOOH and RCO2H are both perfectly acceptable ways to show a carboxylic acid.

The question you raise is a bit of a trick question because the OH oxygen and the C=O oxygen are swapping which is which very rapidly.  This means the answer cannot be tested, for example by using isotopic labeling.

By anology with other carbonyl functional groups, the carbonyl O should be the more nucleophilic.  If it reacts in a nucleophilic substitution, the result is to make the C=O oxygen have a + formal charge when it uses one of its unshared pairs of electrons to make a third bond.  This structure can be stabilized by resonance involving the OH oxygen.  If the OH oxygen reacts as a nucleophile, that oxygen becomes +.  However, since it is an sp3 O, the + cannot be stabilized by resonance.  In addition, the O+ is adjacent to the partially positive carbonyl carbon making it even less stable.  Thus the more stable intermediate, which by considering Hammond's postulate should form faster, is the one where the C=O oxygen acts as the nucleophile