Last night (April 23), Elvis Costello made an appearance on the Late Show as one of David Letterman’s final musical guests before his last episode, which will air on May 20.

As Letterman pointed out in his introduction of the iconic English singer-songwriter, this marked Costello’s 27th time on the show. Costello certainly rose to the occasion, playing a medley of Nick Lowe’s 1980 Rockpile song, “When I Write the Book,” and the singer’s own 1983 classic, “Everyday I Write the Book.” For the performance, Costello was joined by the vocal trio the Quills. Check it out in the video above.

Earlier this month, Letterman revealed the final musical guests who will appear on the show before his departure after 30 years behind the Late Show desk. Mumford and Sons, the Avett Brothers and Brandi Carlile, Emmylou Harris, Tom Waits, Ralph Stanley and Amos Lee, Norah Jones and the Dave Matthews Band are all slated to take the Letterman stage.

As Letterman noted, Costello is scheduled to play the Paramount Theatre in Seattle on April 26. Then, he’ll head to the U.K. between May and June. The singer will return stateside on July 6 in Denver for a tour with Steely Dan.