After discussing David Letterman’s retirement in separate conversations with TV critics James Poniewozik and Aaron Barnhart this weekend, I’m left with a few more thoughts on the subject…

I stopped recording Dave several years ago when it became apparent he was no longer trying to do anything new or interesting. I’d check in on him from time to time, or look at a clip online the next day, but after being a fan of his from his earliest Carson appearances through his NBC daytime show through “Late Night” and “Late Show,” it seemed like Dave just didn’t care anymore.

Neither did I, and I wasn’t alone. Most people would be hard-pressed to name a single thing they remember from the last 5-6 years of his show. If there’s an opposite of viral, that’s what he’d become.

When I think of Dave’s best bits, they’re mostly from his NBC show. Has there ever been a better example of found humor than his visit to a store called Just Bulbs, which sold nothing but light bulbs? The climax of that segment was when he asked the owner what to do if he wanted to buy a lamp shade, and the guy suggested he find a place called Just Shades — followed by a smash cut to Dave standing in front of a store with exactly that name. It was a brilliantly produced bit from the era when Merrill Markoe’s voice helped shape the Letterman legacy and form a new mold for the format. While Dave’s had great moments on CBS, it was during the NBC years that he changed late night television, setting the gold standard with a playbook that Fallon and Kimmel now execute nightly.

In the speculation of who will succeed Dave, you can cross off Jay Leno, Conan O’Brien, Craig Ferguson, and Chelsea Handler. You can also forget Ellen DeGeneres, who has a gig she can continue doing as long as she wants, with a comfortable niche, a huge paycheck, and a bigger audience than any of the late night shows. I’ve also heard Neil Patrick Harris’ name thrown around, but it seems unlikely he’d give up the rest of his acting career — it’s impossible to do Broadway musicals while hosting a nightly TV show.

Jon Stewart isn’t right for the job, either, because we need him right where he is, doing what he does best. Leaving that format — and a remarkable staff of writers and editors — to sit down and interview Chris Evans about the new Captain America movie seems like a shocking waste of time and talent. On the other hand, trying to convert “The Daily Show” to an hour-long “Late Show” wouldn’t work, either. Stewart is already an integral part of the late-night landscape. Moving to CBS wouldn’t be a promotion.

Stephen Colbert’s name has risen to the top of the pack, partly because he works for CBS’ corporate cousin, Comedy Central, and his contract expires at the end of this year, so the timing would be right. Colbert is enormously talented, completely comfortable in front of the camera, and a very good interviewer — even more impressive is how he improvises his way through conversations with his guest while in character. Would he give up his own lucrative show to take over “The Late Show?” If he thinks he’s done all he can do with the character and wants to prove himself on a bigger stage, he will, and he’d be good at it. Colbert also understands how to use (and exploit) social media, an area that both Leno and Letterman never cared about, and it showed, while the younger competition flourished in various online platforms.

There have been lots of comments made about breaking the late night glass ceiling to give a woman a shot. While it’s a good idea to find someone other than a white guy for the job, this isn’t a position for a newcomer. Yes, O’Brien and Fallon got shows without hosting experience, but they were on an hour later in the slot that’s become the training ground for the more-widely-viewed shows. I have a feeling Craig Ferguson won’t continue on “The Late Late Show” much longer, and when he goes, that’s where the door opens for a non-male-caucasian to take a shot.

There’s one debate that’s not worth having, and that’s whether Letterman or Carson was the best-ever. The fact is they were each perfect for their time in TV history. Staying fresh, relevant, and invested in any job for three decades is quite a task, but even more so when you’re charged with entertaining America every night. The truth is that Dave and Johnny’s peak periods ended several years before they called it quits, but that doesn’t diminish their place in the television firmament as legends of late night.

It’s enough to leave it at that.