This looks very much like a memory leak. A correctly running server should have no problems with 4GB of RAM. (That's not to say that more memory might not be useful, but it should be stable in any case.)
You've probably got a tough job ahead of you to track this down, but you can start with some straightforward approaches:
1) When the system becomes unstable, check which processes are using more memory than you'd expect. I would usually set up some perfmon traces to help to get a picture of what's going on in the run up to an out of memory incident. (Once it's occurred, your system is probably not in a good state for diagnostic work either.)
1a) As you have observed that restarting COM+ and the services helps, see if you can get more detail by stopping each in turn and examining your counters again.
2) Review your template code, and if appropriate your mediator, function source, resolver and events code. Check that resource-owning objects run inside using blocks or have a finally block. Check for thread-safety when appropriate.
3) The biggest suspect is your templating code, so try to isolate a specific template. Instead of doing your normal publishing run, publish only one specific page type, but do it a lot. Check your counters and go round. Don't forget dynamic component templates. Once you have a bit of a clue, analyse deeper, perhaps by creating specific test templates/pages.
4) You can sometimes get a result by instrumenting your code with logging calls that dump out the memory at crucial places in your code. The important counter here is the private bytes of the relevant process. A lightweight approach to this is to call OutputDebugString, and pick up the results by using Sysinternals DebugView. Usually a spreadsheet helps too.
5) If these basic techniques fail, you could consider some of the advanced techniques that are available if you are prepared to work with low-level debuggers. This is not for the faint-hearted, but you can do much worse than to start with Tess Ferrandez blog and other published material.
6) Almost forgot... don't forget to check with support whether there are any known issues with your particular version and configuration.
7) Almost forgot again... consider buying some memory profiling software. Typically, you can create an instrumented build and then get an analysis of a test run that points to problematic routines.