Absolutely NOT, Mr Farmer.....You would be saving or as I would say "Liberating" those seeds. If left alone they would blow away in the wind and be lost forever..! Go for it...! ;)
allotment holders would have a different view if it's a site that is being used. Different if it's totally derelict but even then it's still owned by someone.
They are not yours and they are on someone else's plot. Without permission from the plot-holder I'd leave them where they are.
Problem solved I saw yesterday somebody there so problem solved.