Brown in trouble over remarks