In the early 20thcentury, a group of kandar peddlers selling Malay food used to rest on the site where the mosque is now situated. On the suggestion of this group of peddlers, philanthropist and businessman Haji Abdul Khalid bin Haji Mohamed Tyeb decided to turn his land into a wakaf and build a mosque on it. The mosque was then built in 1917 and managed by the Muslim Trust Fund Association (MTFA) until 1968.
The mosque has kept its original structure with minor alterations to accommodate its growing congregation. In 1996, major renovations were carried out to restore the building to its present splendour. The first Imam of the mosque was Almarhum Syeikh Osman bin Syaikh Omar Al-Ganus, an Arab from Palembang. He was also a famous Kadi (registrar of Muslim marriages) in Singapore at that time. Almarhum Syeikh Osman was also a great cook and the tradition of preparing food at the mosque still lives on today. This has made the mosque famous for good food that is available there during occasions on the Islamic calendar.