One should definitely make a visit to Morgenstern's Finest Ice Cream on Rivington Street. It's a late-night spot with a impressive range of flavors. Although not 24 hours, they do remain open until midnight which ideally meets your prerequisites.