The only true "public" kindergartens here are the PAP kindergartens which are located all over the island, but even those have a cost to them, a sliding scale based on income - however, being a foreigner you'll probably not get any subsidy. The next step up are the church affiliated kindergartens which generally cost around $300 per month regardless of whether you are local or a foreigner. Then there are the private "expat" kindergartens which range $600-1,200 per month. Most
International Schools have expensive kindergarten programs as well.
Please note that if you intend to send your child to local primary school, you will most certainly want to send them to a local kindergarten so they are better prepared. I recommend the church affiliated kindergartens which tend to be stronger in English and they almost all teach Chinese as a second language.
After trying many kindergartens here we eventually stuck with Barker Road Methodist. My daughter did 3 years there and my son is doing his second year there. Although no school is perfect, Barker Road is less rigid (by local standards), and our kids have been happy there. My daughter had an easy transition to local primary school this year.
Good Luck!