If you want to build one of these I'd suggest getting in touch with
Austria Micro Systems and see how much it would cost for a previous gen ARM chip like they used in the Sansa Fuze or Sansa Clip.
These devices are supported by Rockbox and so you have a wealth of ready to use open source code and ready to go dev tools. The clip already has a minimal display but in either case you would reduce the device to just the chip, one line display and button. No battery, no wheel or sound or other stuff. Just chip, usb connector, display, button on a pcb. Your main cost is the AMS chip and I don't know what the volume price on those is but I'm sure it's capable enough to do the needed crypto functions, unlike the cheap 8 bit PICS and AVRs etc. One line LCD display can be very cheap. I bought mobile phone LCD displays for $2-3 and they were multi-line like on Nokia phones.
You might also look at some of the Broadcom chips available like they use in the Raspberry PI, though you could get by with one much less capable than they use for that. I don't know what the cheapest ARM chips on the market - you should search for the lowest priced one as you don't need much more than cpu+usb+some flash memory.
The nice thing about using the AMS chips is you can buy a second hand Fuze off ebay and do all the dev work and hack in a display and test it fully before you even make a PCB and order anything. So you could do a prototype for almost nothing and look for funding to do a real production run.
BTW I contributed to the Rockbox project and did dev work on my Fuze so I know all this to be doable quite easily. Heck, you could probably just market the open source Sansa Fuze/Clip compatible code to make this thing work with the millions of Clips/Fuzes kicking around now. It's pretty easy for an average user to upgrade the firmware too and it can become a Bitcoin wallet device.
RockCoin ... Bitcoin Wallet for Rockbox. Solid Like A Rock. Or
CoinBox sounds better.
Fuzes with broken display are dumped on ebay for like $5-10 or so. Or they were, I haven't checked recently.
Edit: Just saw on AMS web site 1k qty chip is $9 each. A bit high but maybe they have a lower end chip available. USB conn stk#10019 from4uconnector.com about 0.13 each, PCB 0.5"x2" from China low qty about 0.50 each. May need a volt reg if not already on chip, 5V > 3.3V likely.