It doesn't take 5 hours to get to Yosemite Valley from SF. It's more like 3.5. I'm actually about to do it tomorrow morning! Also, driving to the Agnew trailhead is impossible unless you arrive in Mammoth the day before---you have to pick up a permit from the ranger and they open later than the road closes. As far as what's the best route to hike, it kinda depends on what you like. The River Trail in the valley there is heavily forested and monotonous while the High Trail is exposed, out in the open, with major megaviews the whole way basically (can you tell what I like? )
Depending on exactly how many days you have, I'd suggest something like what's been suggested before:
Day 1: Leave SF as early as possible. If you leave by 6am or so, you'll be in Yosemite by 10am latest. (You're headed opposite of commute traffic. Yay!) I'd head in via the 140 and check out the Mariposa Grove and Glacier Point before driving down into the valley. Drive around the valley with one million billion other people who booked their trips five months ago and are super bummed about the small waterfalls this year due to 30% of average snowpack. Don't bother hiking anywhere, but do see what you can see from the valley and pullouts. Around 4pm, head out of the valley and drive along Tioga Road to the east. This will give you enough light to see some stuff as you drive east. Head to Mammoth Lakes and camp in one of the Shady Rest campgrounds.
Day 2: Pick up your permit from the ranger in the morning and take the shuttle to the Agnew trailhead. Hike the High Trail in to Thousand Island lake. Camp. The sunrise and sunsets at Thousand Island are super nice. Then hike out however you want. If start early and take the JMT to Shadow Lake to the River trail to Agnew (about 8-9 miles), I think you can be back at your car in time to get a couple hours of driving in before shacking up at a motel in BFE.
Day 3: Drive to Zion. You can probably be there by early afternoon, so you have time to check some stuff out around the canyon and sign up for everything for the Narrows trip the next day. It's an amazing, truly unique, experience. I did it as an overnight 8 or so years back. Backpacking up the West Rim trail is super rad also.
This is great info, especially about what kind of scenery we should expect along the High Trail and River Trail. Do you know if the JMT in that area is more similar to the High Trail (exposed) or River Trail (forested)? I do want to use the JMT either going in or coming out, so I'd probably decide on either the High Trail or River Trail based on what was going to give us the most diverse experience.