Ripples in Space-Time Reveal Traces of Black Holes From the Beginning of Time