I am going to answer my own question, as I got a lot of additional information from other sources and think its helpful to summarize it here.
It is not at all possible to use Session Preview without the Ambient Data Framework as key to this working is the ability for your web application to switch between getting content/data from the normal staging storage (broker/DB) and the Session Preview storage (a separate broker DB) and merging the two if required.
This process is handled by two servlet filters (J2EE) /HTTP Modules (.NET) (the ADF one, and a session preview one), so you need to have this active on the environment serving your content.
You can have your content served through the SDL Tridion CD Webservice with this process enabled, which gives you the option of moving ADF elements from within your web application into the webservice (as Daniel mentioned, you need to pass the Preview Session token to the WS call). This however is not fully featured in SDL Tridion 2011 SP1 (again as Daniel mentioned in the comments you cannot merge query results from session and staging storage), but will be fully featured in the SDL Tridion 2013 release.
If your pages are on the filesystem, they are not available though the CD webservice (all items available through this must be in the broker database), so you cannot use the ADF on page requests (and thus Session Preview) unless the ADF is within your web application.
So in summary, can you use Session Preview without the ADF? No, but you can remove the ADF from your web application by using the CD Webservice with the ADF and calling this from your web app. In this case however Pages (and other items) should be published to the broker DB and if you are using SDL Tridion 2011 SP1, you may be limited in the behaviour of preview on query results.