I have encountered a similar problem with this species; I went to see them in the very dry year of 2011 when I went at a similar time of year. They had virtually all gone over by that time and only a few plants could be found in flower tucked away underneath the tall bracken.
I found a total of 4 plants (in seed) at the site you went to but the best population is on the verge near Pownsley Wood which is only a few miles away. The best time to go is early June in anything but a very wet or cold year, the flowers go over very quickly once they have been pollinated.
I can get a grid reference for you for next year's visit tomorrow. Alternatively, try the Plantlife webpage as they did a survey for this species in 2011 (I noted the training day that year was something like the 7th June).