If you're interested in having such a device designed, I'd be interested in contracting.
Primary plan of attack would be a PIC32 doing USB HID+MSD with an EEPROM. Nothing revolutionary.
The PIC32 toolchain isn't entirely open, but the compiler is open and all required tools are cross platform. The PicKit3 programmer is available for $45. Clones exist if you want to save money. (IMHO they're not worth the trouble, but it's your prerogative.)
The parts list would look something like this:
uC:
http://www.digikey.com/product-detail/en/PIC32MX230F064B-I%2FML/PIC32MX230F064B-I%2FML-ND/3046648Display:
http://www.digikey.com/product-detail/en/NHD-12232KZ-NSW-BBW-P/NHD-12232KZ-NSW-BBW-P-ND/1701265Serial EEPROM (multiple for redundancy?)
Buttons:
http://www.digikey.com/product-detail/en/MJTP1138BTR/679-2415-1-ND/2344168 (A shorter button would be used if you want keycaps)
USB connector:
http://www.digikey.com/product-detail/en/48037-0001/WM17117-ND/857603Estimated BOM price is about $25 with PCBs.
Potential issues:
Security. Are you looking for encryption and authentication? How about hardware security?
Size. The display is going to make a big difference here. What content do you want to display?
Case. I can 3d print cases, but they're neither as pretty nor as durable as injection molded cases.
Write only. A tamper proof write only setting would take some extra effort. How permanent does this need to be?
What's your desired capacity?
Software. USB software is generally painful to develop.
E: \/ John, that post made me search around for an upvote button. Looks like a great product!