Who feels the kick 1ms afterwards is closest to the kick since the sound took the least amount of time to travel to him. The person that felt the kick 4ms afterwards is the furthest away from the kick. You are somewhere in between since the time it took the sound to reach you is somewhere in between. Taking more time to travel to travel to an individual corresponds with an increase in the distance that it takes to travel to the individuals. Not only that, but also a proportional increase in the distance since sound travels at a constant rate.
I have in this attached file a drawing illustrating how the people would be arranged and what distance they would be from the source of the sound.