Migrations are always a big topic for debate, while in my humble opinion they never have to be. Please let me start of with mentioning my blogpost about content migration and migration tools: http://www.tridiondeveloper.com/how-to-say-goodbye-to-your-migration-tool
In here I explain (based on my own experience), there is no better way to migrate content than to do it manually. When you hire so called cheap labor (college students, etc.) it will cost you less and give you a higher rate of success. Plus, who wouldn't prefer a blond 21 year old sitting across from you doing all the work, over an old crappy computer which is blowing dust and burning CPU cycles?
So to answer your question directly I'll list your options in my order of preference:
Manual content migration (cheap and fail-safe)
Database migration (see my answer heremy answer here)
Content Porter (see my answer heremy answer here)
Custom built migration tool
In case you really want to go down the route of option #4, Then there are two options available basically, either you create a script (using VBScript TOM API) or VB6 program that exports the content on your R5 system, or you use the R5 Business Connector SOAP API for that. As for importing, you will have to create an import client using the 2011 Core Service API (as you can see you will have to do a lot of work, realistically I think its safe to say, option #1 will be cheaper).