The date of the Gondor's loss of Umbar is not recorded. All we know for sure, from Appendix A to The Lord of the Rings, is that '...in the evils that soon befell Gondor Umbar was again lost, and fell into the hands of the Men of Harad'. The 'evils that soon befell' were the waves of assault on Gondor, especially by the Wainriders, that started in III 1856.

The fact that Umbar 'fell into the hands' of the Haradrim seems to suggest that Gondor voluntarily withdrew its occupying force to meet the Wainrider threat, rather than losing it to military action. This is open to question, but what seems clear is that the Haradrim permitted, and indeed probably encouraged, the Corsairs to resume their raids on Gondor's coasts.

The last we hear of the Corsairs is on 13 March III 3019, the date that Aragorn captured their fleet and sailed it to the aid of Minas Tirith. This was not their first defeat at Aragorn's hands. Nearly forty years earlier, in his guise of Thorongil, he led a devastating Gondorian attack on Umbar that left the Corsairs' fleets all but destroyed. After these two great defeats, and given the growth in Gondor's power after the War of the Ring, it seems unlikely the Corsairs survived long into the Fourth Age.