Assuming that your allotment site is council owned I would contact them.
Yes council or kill them as fast as possible. Use some bug killer spray.
Wasps will only be a danger about September time when they "Go Mad" if you dont want to wait for that pleasure then just get rid before then.
Got it sorted now thanks for all the info peeps