The first known owner of the Liubavas Manor was the Goštautai, the most influential and powerful Lithuanian noble family in early 16th century. Albertas Goštautas (Olbracht Gasztołd) was married to princess Zofia Wierejska (1485–1549(51)).
Albertas Goštautas held important posts in the Lithuanian state and military service. From 1503 to 1509 he was the Grand Duke's Cup-Bearer of Lithuania (cześnik litewski) and the vicegerent of Navahrudak. He participated in many successful military operations: in 1506 he withstood the attack of Crimean Tatars, in 1507 defended Smolenks from the army of the Grand Duchy of Moscow, a year later organised a successful protection of the treasury of Grand Duke of Lithuania Žygimantas Senasis (Sigimund the Old) from conspirators of Mykolas Glinskis (Michał Gliński). From 1513 Albertas Goštautas was Bielsk starost, Mazyr starost from 1515, voivode of Navahrudak from 1511 to 1514, voivode of Polotsk from 1513, voivode of Trakai from 1519, governor of Darsūniškis from 1523, and Barysaw starost from 1525. In 1522, the nobleman became the voivode of Vilnius and the Grand Chancellor of Lithuania. He was granted the title of the Geranainiai count by the Charles V, Holy Roman Emperor.
Albertas Goštautas is definitely one of the most prominent and important Lithuanian historical figures of all times. Even though he held various state posts, he supervised the drafting of the First Statute of Lithuania. He was a man who cared about the future of the State of Lithuania. He looked deeper and much further than many of his contemporaries. Albertas Goštautas highly valued the independence of Lithuania. It is very likely that, seeing threats to the statehood, he demanded to banish Polish Bernardine monks who did not speak Lithuanian. In 1530, he obtained a separate Lithuanian Bernardine province from Pope Clement VII. On the initiative of Albertas Goštautas, the Statute of Lithuania included provisions prohibiting aliens, i.e. the Poles, buy Lithuanian assets or hold offices. Protecting the independence of Lithuania, Albertas Goštautas managed to make sure that minor Žygimantas Augustas would be crowned the Grand Duke of Lithuania in 1529, even though his father Žygimantas Senasis was still alive. Albertas Goštautas supported attempts of Duke of Prussia Albrechtas Brandenburgietis (Albrecht von Brandenburg-Ansbach) to become a Polish regent after the death of Žygimantas Senasis. Unfortunately, he did not succeed and in 1530 the Polish Sejm elected Žygimantas Augustas the King of Poland. This caused conflicts between mother of Žygimantas Augustas Bona Sforza, who wanted to reduce the influence of the Goštautai on the state, and Albertas Goštautas, who wanted to limit the power of Bona in Lithuania.
After Albertas Goštautas died in 1539, the Liubavas Manor was inherited by his son Stanislovas Goštautas (Stanisław, 1507–1542). Together with Liubavas, Stanislovas also inherited other manors: patrimony Geranainiai, Kėdainiai, Liubčia, Mėguva, Biržai, Trakai, Medininkai, Dieveniškės, Tikocinas, etc. The career of Stanislovas Goštautas started at a very early age. In 1522, at the age of fifteen he was appointed by the Lithuanian Seimas the voivode of Navahrudak. He then became the vicegerent of Navahrudak from 1530, and the Voivode of Trakai from 1542. He was also a member of the Lithuanian Council of Lords. Stanislovas Goštautas, like his father, was an extraordinary personality. He is thought to be the initiator of the Lithuanian yearbook, the so-called Bychowiec Chronicle, patronised Mykolas Lietuvis.
On 17(18) May 1537 Stanislovas Goštautas married Barbora Radvilaitė (Barbara Radziwiłł, 06-12-1520–08-05-1551). He died on 18 December 1542, likely in Geranainiai where he used to reside, without leaving any descendants. According to historical sources, soon after the Liubavas Manor, like other large estates of the Goštautai family, was transferred to the state under the ishmara law, and the larger part of the property to the Radvilos family through auctions of the assets. Sofija Goštautienė, mother of Stanislovas Goštautas, could still use the Liubavas Manor for some time (she died around 1549(51)). It was the end of the Goštautai dynasty, the most influential Lithuanian noblemen and counts in the 15th and 16th centuries.
Ten years later, in autumn 1547 widow Barbora Radvilaitė Goštautienė secretly married Žygimantas Augustas, last member of the Jagiellonian family, Grand Duke of Lithuania, King of Poland and owner of the Liubavas Manor. A little bit later, Žygimantas Augustas exchanged manors with wife's brother Mikalojus Radvila the Red (Mikołaj Radziwiłł Rudy). He traded Liubavas for Rokantiškės.