You're in luck, I was in this exact position last August, looking at RS3, S4, GTR and 911 Turbo.
I disregarded the last 2 quickly after test driving and seeing my little girl squashed up in the back
Didn't consider the RS6 as I had an E60 M5 a few years ago when they first came out and hated the consumption.
My thoughts, the S4 is a good solid rep mobile, and will suit most, but if you want something fast and special then it's the RS3 all day long!
S4 has a nice cabin, looks nice, and is reasonably quick. Prices plummet though, and without tuning it and invalidating any warranty it's never going to be an exhilarating drive. The S4 is a compromise to an RS and there is no denying it.
The RS3 is a superbly packaged car, the haters (see DK) will say it is dated etc etc but it is still the fastest, best looking A3 to date! To 100mph there isn't much on the road that will beat it, it handles superb, has an engine note to die for and has bullet proof residuals.
I've had many, many cars like this over the years and am always looking to change after 6 months due to boredom, the RS3 is the first car I have had for over 6 months and I'm actually looking forward to the good weather so I can get some fun miles on the clock
I can honestly say there is NOTHING out there without getting in to serious debt that would replace it.
You will test drive both today, I'm confident you'll make the right choice, the S4 is such a bore to drive very much like a 330i is to an M3.
Enjoy your day though!