You can do something like:

    var taxonomyFactory = new TaxonomyFactory();
    var filter = new CompositeFilter();
    // all keywords 
    filter.DepthFiltering(DepthFilter.UnlimitedDepth, DepthFilter.FilterDown);
    // Creates a flat list
    TaxonomyFormatter formatter = new TaxonomyListFormatter();
    Keyword keywords = taxonomyFactory.GetTaxonomyKeywords(categoryId, filter, formatter);
    IList keywordList = keywords.KeywordChildren; // list of Tridion.ContentDelivery.Taxonomies.Keyword 

Do note that when retrieving a flat list like this, you will lose all the information about hierarchy in the Taxonomy, which means that if you want to get the parent keyword of a keyword, you need to fetch that keyword again.