I notice that when session preview is enabled for Experience Manager, my page is auto-published to my staging website when I click Finish Editing. When I disable session preview, it no longer auto publishes. Is it possible to configure this behaviour somewhere (for example if I do not want to auto-publish when using session preview, or do wish to auto-publish when not using session preview)?
No, there's no configuration for that.
However, the behavior is unrelated to Session Preview -- it will always attempt to Publish the item after it has been saved successfully. I'm not sure what scenario you are describing where it doesn't Publish - but maybe it was because there were no changes to save?