An important aspect of geometric group theory concerns classifying groups and metric spaces up to quasi-isometry (that is coarse geometric equivalence). A related aspect is describing self-quasi-isometries. A recent result of Behrstock, Kleiner, Minksy and Mosher shows that the mapping class group of most compact surfaces are quasi-isometrically rigid: that is, any quasi-isometry is induced up to bounded distance by a homeomorphism of the surface. We describe how this and related results can be viewed in terms of a coarse median structure, (an idea inspired by work of Behrstock and Minsky). We obtain some variations and generalisations of this result.