This delightful multi sports venue on the south coast was, and still is, part of the Duke of Devonshire’s estate, where in Victorian times the most expensive spice in the world was harvested. Neville Johnson visited the eponymous Saffrons to find out more about it, with a surprising discovery