This is a very broad question, that cannot really be answered without knowing what your templates are doing.
There's a few things to look at. You report stability issues, which would suggest you have code quality issues. Perhaps bad memory management, usage of COM objects in templates that are not correctly released from memory, etc.
You also should look into where the time is being used. If you double click on any successful transaction in the publishing queue you'll be shown details of that transaction, like how much time is spent on which phase - resolving, rendering, transporting, deploying. Based on this info you'll know where you should focus. Likely, most time will be in rendering, and therefore you need to dig into what your templates are doing.
There are things you can do to speed up publishing time from a server config point of view, like playing with the publisher threads (more threads does not mean faster, means more tasks in parallel, which may slow down the database) and with cache settings for the publisher too. But you'll never get as much benefits from this than from reviewing what the code in the templates is doing, as that is what is using CPU cycles and memory.
So, my top recommendation would be to have an audit of your templating code. Possibly you're spending time reading the same information over & over when it's not needed and it could easily be cached. Perhaps you're creating static references to objects that can't be used in a static manner, like the Engine and Package. Perhaps you're loading data from a virtual folder, which means that the results can't be cached by the CM. All of this can only be determined by looking at the code.