Possible issue with XPM
The biggest issue I've had with multimedia for CSS is having the CSS blocked in IE for Experience Manager. I'm not sure it was a browser security or configuration problem, but be sure to check these binaries are delivered with the right mime type.
Aside from performance considerations that the other address, you also have to consider how you will publish these. Since they're in metadata,
addbinary() or publishing them independently (with their "dynamic" component templates) are your main options (I don't think metadata links are automatically resolved, but check "items to publish" to be sure). I describe the basic publishing propogation scenarios in this post.
Confirm logic and handle duplicates
John Winter actually taught me a nice scenario that looks for, and removes duplicate CSS files, at different contexts (the queued page, template metadata, structure group metadata, etc) before adding a consolidated set to the rendered page. You may want to confirm what to do with duplicates in this sense as well as same-named multimedia files (which isn't necessarily a unique problem to your approach).
Functionally, I like the (recursive) fall-backs that come with this approach. You can quickly control a website section (SG) but have flexibility at the page level. Just be wary of abuse. :-)