Is the number of coin control inputs going to matter?
Yes, every input in a transaction matters. The more the number of inputs you add to your transaction, the more the size becomes bigger. Let me make an illustration for you:
Let's say our basis for inputs and output are native segwit(version 0) since you received your coins from signature, the transaction input is 68 vbyte and 31 vbytes for output.
If you are spending:
1 input and 1 output will give you 109.5 vbytes transaction.
2 inputs and 1 output will give you 177.25 vbytes transaction.
5 inputs and 1 output will give you 380.5 vbytes transaction.
More input and 1 input will give you bigger vbytes transaction consequently.
Now, imagine having 10 or even 15 inputs for a single transaction, you will have to pay more fees for every vbytes for that transaction. More inputs, more fees.
You can check this
thread by Charles-Tim for more understanding.
However, with coin control, if you have much inputs and you want to spend only few Bitcoin, let's say you have $300 worth in total balance and you want to spend $100, you can just add few inputs that can give you equivalent $100 or more to spend and get back a change if there is from your transaction.
is it going to affect the transaction speed or its just a matter of privacy?
Speed of transactions has nothing to do with your inputs, if you want a faster transactions, you have to pay an optimal fee to get your transaction included in the next block.
Privacy comes when you spend inputs together.