In SDL Tridion 2011, Inside a Component is it possible to get dynamic behavior on the controls?
Like if user selects s particular optional button then only user is able to select the other dropdown values, else the dropdown box will be hidden?
This functionality is not currently available, What about if you manage this dynamic controls logic in a Window outside tridion like a Custom URL and just pass the values to a Component Field after you have selected all the right values from those dynamic selection controls?
In the component schema definition you can specify a Custom URL that is executed when you click in a component field label, it will open a new window (you will need to develop that window) with dynamic generated selection controls.
We have a few out-of-the-box setups to control subsequent selections in Tridion:
In the first scenario, authors select the first value among a set of keywords, then "drill-down" to options relevant to each selection. Component links make sense when the fields should be used and managed separately. If the content fields differ that much, consider separate schemas.
After these, I'd recommend The custom url approach Eric mentions. I'd only recommend a GUI extension if you can account for all current and future schemas as well as maintain the setup. But at that point you're looking at building something like InfoPath on top of the CME.
Schemas are just XML schemas and the UI is generated from that. The validation options available are the field type and whether the field is mandatory.
If you want to add custom behaviour such as this I think the best way would be to write a GUI extension. An uglier option would be to add Event system code to validate these rules but I would recommend a GUI extension to achieve what you describe.
Otherwise the best (quickest) option is to document rules like this in a content guide or wiki.