This knowledge is not conventional knowledge which is limited, and subject to mistakes and being forgotten. This knowledge is given by God. The Prophet (s) says that regarding such servants, God told him:

I will love them when they love Me and I will make them loved by My creation. And I will open up their inward eyes to My Glory and Grandeur and I will not hide from them [the knowledge of] the select of My creation. In the darkness of night and the light of day, I will tell them secrets so that their conversations with creatures and with their companions will be cut off. I will make them hear My words and the words of My angels. And I will reveal to them the secret I have
hidden from My creation.3

Such a person is inspired by God so much, and receives communications from Him, that they do not need to ask anyone about anything or speak to anyone. Therefore, their conversations with others are kept to a minimum. This means that either they only speak when necessary, or it can mean that even when they are talking and listening to people they are in constant communication with God, listening for God’s message to them in the form of words coming from other people. We can receive messages from God through the words of people. For example, sometimes a child may give us a message which is from God even though they themselves do not know this. Sometimes even our enemies may tell us something which is very useful and thus it is a message for us from God coming through that channel. Therefore, it is possible that the hadith means that such people will keep their conversations with people to a minimum or that they would effectively be listening to God all the time, even when engaged in conversation with people. Indeed it can actually mean that both things are true.

3Irshād al-Qulūb ila al-Sawāb, vol. 1, p. 204. The original text is as follows:
فَإِذَا أَحَبَّنِي أَحْبَبْتُهُ وَ أَفْتَحُ عَيْنَ قَلْبِهِ إِلَى جَاللِي فَالأُخْفِي عَلَيْهِ خَاصَّةَ خَلْقِي
فَأُنَاجِيهِ فِي ظُلَمِ اللَّيْلِ وَ نُورِ النَّهَارِ حَتَّى يَنْقَطِعَ حَدِيثُهُ مِنَ الْمَخْلُوقِينَ وَ
مُجَالَسَتُهُ مَعَهُمْ وَ أُسْمِعُهُ كَالمِي وَ كَالمَ مَالئِکَتِي وَ أُعَرِّفُهُ السِّرَّ الَّذِي سَتَرْتُهُ
عَنْ خَلْقِ