The Yap Kongsi was established by the Yap clan, one of the oldest and most prominent Hokkien clans in Penang. The clan traces its lineage back to 439 BC in China, originating from warrior Shen Zhu Liang. In Penang, two separate Yap associations—the Tong Eng Siah Kongsi and Hooi Teik Choon Ong Yap Kongsi—merged in the early 20th century to form the Lum Yeong Tong Yap Kongsi.
The present building and temple were completed in 1924 on land donated by the local tycoon and clansman Yeap Chor Ee. The temple section, Choo Chay Keong, is dedicated to the patron deity Hoay Che Chun Wang (also known as Tua Sai Yah) and houses the ancestral tablets of the Yap family.