Use a hard plastic mallet. If it's too loud, you can always put a bandana or towel over the top of the brake drum to help take the volume down without sacrificing a whole lot of tone.