Beneath a highway overpass, but don't expect room service... Seriously, you have to be a bit more specific as to what you're willing to spend, what you're willing to tolerate, how long you'll stay and at what point of the year. What is cheap in Central is not as the same as cheap in Yuen Long. There are obvious trade offs in time and daily travel expenses from each location too. If you're going to stay for a long time, try the service apartments (Google service apartment hong kong and you'll get hundreds of hits). The cheapest place to stay of course, is with a relative. But you'll generally have to put up with their company. -lol
I remember staying at a hotel near the airport. It was pretty nice, but it cost $800 a night lol. Like ralph said, be more specific.
How long are you planning to stay in HK? If it is long, you can try service apartments, they are cheaper than hotels, since the price is not per day. But per week or month.