It all depends to what event you actually subscribed, there are three parameters when you subscribe to an event, the subject type, event type and the phase. In your situation, the event type is crucial.
When you use the
Tridion.ContentManager.Extensibility.Events.PublishOrUnPublishEventArgs, you will subscribe to both the Publish and UnPublish actions, details of which you should be able to find in the
ContextVariables property of the EventArgs.
However, you can also subscribe separately to the
Tridion.ContentManager.Extensibility.Events.UnPublishEventArgs, these are probably the ones you are looking for, so you can separate your code even more cleanly. If you do require some code to be run on both the Publish and the UnPublish event and other code on either one of those separately, then you will want to debug your event system and check out what the property
ContextVariables of your EventArgs contains.
As you will notice, the API Guide (requires login) is your friend here, just make sure you zoom in on the classes to see the full inheritance hierarchy.