Originally established in 1748 by Indian Muslim traders, the mosque was first a simple wooden structure. It was later rebuilt in brick and concrete in 1872 while meticulously preserving its original eclectic design. The mosque derives its name from the surrounding area, Kampung Kling, which was once the residential quarter for the South Indian merchant community. The mosque is a physical manifestation of Melaka's history as a global trading hub, incorporating Malay, Chinese, Hindu, and European elements. Its location on Jalan Tukang Emas (Goldsmith Street), better known as "Harmony Street," is highly symbolic. It sits just steps away from the Sri Poyatha Moorthi Temple (Hindu) and the Cheng Hoon Teng Temple (Chinese), serving as a living symbol of religious pluralism and peaceful coexistence in Malaysia.