Would it also work if I started out without the pruned flag and added it later (culling disk space effectively)?
What about the case of a non-pruned node that's running with neither flag and then I restart it with block filters?
Hence, your block filter should be present after your IBD. You can prune after you've synchronized fully, or you can prune as you go. If you have a non-pruned node without block filter but decide to enable after, then you need to reindex as well to build it. Note that blockfilterindex provides for the filter and the peerblockfilters allows your peers know that you're able to service BIP157 and serve them the filters.