Author

Topic: Verifiable builds need attention. Only 3 of 68 Android wallets are verifiable (Read 587 times)

full member
Activity: 1092
Merit: 227
First of all thank you for the great work you are doing with the wallet scrutiny, I just stumbled upon your thread while I was reading through this section. It is amazing how you started your journey since 2019 and have already sirupted lot of information about various wallets. I think CoinBase is now even worst at this point. The user base is has outgrown the previously published number which means there are more and more users who just want to use nice looking apps, with easy to handle request. I mean pick a wallet which is non custodial, they have basic UI, (though its advancing) people are not used to it somehow. May be they like to share their private keys with the custodial wallets. Sadly the number of such wallets is huge which makes us think there is such under education about the "Not your keys, not your bitcoin".

So because they point out that a custodial wallet is custodial, that custodial wallet attempted frivolous legal action because they didn't like someone pointing out that they are custodial? Roll Eyes
We should clearly name those custodial wallets and tell more people to avoid using them.
WalletScrutiny received this legal actions two times so far, first time it was from Mercado Bitcoin, and now they received it from Foxbit.
Both of this services are located in Brazil, that makes me think there is some connection between them, and this was targeted attack.

I am sure this is targeted attack only. They are just trying to take down the information and do not want to get hampered with their brand. This is what makes them afraid. Since you are openly stating the information about the custodial wallets they would lose the client base. Not sure how you are going to fight back but definitely under information acts this isn't illegal. Good luck.
legendary
Activity: 1568
Merit: 6660
bitcoincleanup.com / bitmixlist.org
MercadoBitcoin's review was removed due to a DMCA takedown notice and I had forgotten to put it back up until the Foxbit takedown notice.

Today the latter just had its deadline to take court action expired, so I will re-instantiate it, too.

Excuse me, but what the fuck is going on? Are some companies apparently angry at your website or something? Huh
legendary
Activity: 1862
Merit: 1114
WalletScrutiny.com
MercadoBitcoin's review was removed due to a DMCA takedown notice and I had forgotten to put it back up until the Foxbit takedown notice.

Today the latter just had its deadline to take court action expired, so I will re-instantiate it, too.
legendary
Activity: 2212
Merit: 7064
So because they point out that a custodial wallet is custodial, that custodial wallet attempted frivolous legal action because they didn't like someone pointing out that they are custodial? Roll Eyes
We should clearly name those custodial wallets and tell more people to avoid using them.
WalletScrutiny received this legal actions two times so far, first time it was from Mercado Bitcoin, and now they received it from Foxbit.
Both of this services are located in Brazil, that makes me think there is some connection between them, and this was targeted attack.
hero member
Activity: 406
Merit: 443
WalletScrutiny website was recently shut down due to a DMCA takedown notice, and we it was unavailable for few days but now it's working again.
I don't know why anyone would have a problem with website like this, but maybe some closed source wallets and centralized exchanges are involved with this.
If you want to follow what happened follow their twitter account @WalletScrutiny.
The power of these "custodial exchanges" and the time that a lawyer took to litigation and all this for open source code?
I will download the website files, run it offline, and will everywhere give Foxbit (which was the reason for DCMA) a negative rating. Cool

187 wallet with No Source! and 591 Custodial! All this in a decentralized industry.
legendary
Activity: 2268
Merit: 18711
So because they point out that a custodial wallet is custodial, that custodial wallet attempted frivolous legal action because they didn't like someone pointing out that they are custodial? Roll Eyes

What incredibly scummy and shady behavior, but unfortunately anyone who is already using a custodial wallet is unlikely to swayed by said behavior.
legendary
Activity: 2212
Merit: 7064
WalletScrutiny website was recently shut down due to a DMCA takedown notice, and we it was unavailable for few days but now it's working again.
I don't know why anyone would have a problem with website like this, but maybe some closed source wallets and centralized exchanges are involved with this.
If you want to follow what happened follow their twitter account @WalletScrutiny.
legendary
Activity: 1862
Merit: 1114
WalletScrutiny.com
Some people also reported that NVK blocked them on twitter so they can't comment on any of coldcard twitter posts, and he is hating all other hardware wallet devices...

#metoo
legendary
Activity: 2212
Merit: 7064
Cringy :/ I used the term "open source" in a sloppy way, too but once somebody complained, I replaced it everywhere with "public source", as that is what I need to reproduce binaries.
I don't know why NVK is pushing this open source label so hard, but it's crystal clear that you can't have common clause license in your code and still advertise code of your device as open source.
Some people also reported that NVK blocked them on twitter so they can't comment on any of coldcard twitter posts, and he is hating all other hardware wallet devices...
Quote
Is this “Open Source”?
No.
https://commonsclause.com/
legendary
Activity: 1862
Merit: 1114
WalletScrutiny.com
btw ColdCard wallet is not open source anymore, but he still claims differently on his website... maybe he will change that in future also Smiley

Cringy :/ I used the term "open source" in a sloppy way, too but once somebody complained, I replaced it everywhere with "public source", as that is what I need to reproduce binaries.

As long as the site is not created to advertise certain project(s) and as long as they are giving correct information instead of falsified claims, things can remain healthy otherwise that's another case of why centralized "review" sites are generally bad.

WalletScrutiny is certainly not created to advertise any specific products. I did work for Mycelium before but I quit because of this conflict of interest and I'm pretty open about my disagreements in direction when it comes to shitcoins. Ideally WS would be easy to fork though but I don't dare yet to make the reviews themselves creative commons or something. The framework and tools are open source already though.
legendary
Activity: 3472
Merit: 10611
I don't think there is anything wrong with some (healthy) competition and it can help improve both of your websites.
As long as the site is not created to advertise certain project(s) and as long as they are giving correct information instead of falsified claims, things can remain healthy otherwise that's another case of why centralized "review" sites are generally bad.
legendary
Activity: 2212
Merit: 7064
I feel like bitcoinbinary was launched as a reaction to WalletScrutiny's review of ColdCard. It's @NVK's project.
Yeah it's his project, and he even ''donated'' himself, but it appears he now removed that info from the page bottom (maybe because of my remarks), however it was archived on time Coinkite = Coldcard = NVK Smiley
I don't think there is anything wrong with some (healthy) competition and it can help improve both of your websites.

btw ColdCard wallet is not open source anymore, but he still claims differently on his website... maybe he will change that in future also Smiley


https://coldcard.com/
legendary
Activity: 1862
Merit: 1114
WalletScrutiny.com
I feel like bitcoinbinary was launched as a reaction to WalletScrutiny's review of ColdCard. It's @NVK's project.
legendary
Activity: 2212
Merit: 7064
There is one alternative website I found for WalletScrutiny and it is called bitcoinbinary.org, interesting part is that one of bitcointalk moderators achow101 was testing wallets and participating in this exercise,
I don't know if this website is sponsored by Coinkite aka Coldcard, but they did receive 0.025 BTC donation from them and githuib page is posted on Coinkite github,
so it looks like ColdCard wanted to proved how their code is still reproducible even if it's not open source anymore.
Conclusion is that many wallets have bad documentation or incorrect build instructions so they couldn't be reproduced.


Github: https://github.com/coinkite/bitcoinbinary.org
Website: https://bitcoinbinary.org/

legendary
Activity: 1862
Merit: 1114
WalletScrutiny.com
...
How exactly are you testing Hardware Wallets?
I guess you first need to have actual device in your hand (purchased or received for testing from manufacturer) and then try to reproduce the code.

We look at claims about the functionality of the device to see if it falls into any of the k.o. criteria like not having a screen to verify what you approve. Then we look for the source code and the binary. If the source code compiles into the binary, the wallet is reproducible. Check out our full methodolgy.

So ... if you want to help, there is a ton to do from simple triage to compilation to design to spreading the word. Wink
I am helping in spreading the word about WalletScrutiny and I am monitoring hardware wallet changes, especially if they claim they are open source.
You can track that in my topics that is updated on regular basis like this one for example: LIST - Open Source Hardware Wallets.

I think we have all the products you list. We have to review most of them still.
legendary
Activity: 2212
Merit: 7064
...
How exactly are you testing Hardware Wallets?
I guess you first need to have actual device in your hand (purchased or received for testing from manufacturer) and then try to reproduce the code.

So ... if you want to help, there is a ton to do from simple triage to compilation to design to spreading the word. Wink
I am helping in spreading the word about WalletScrutiny and I am monitoring hardware wallet changes, especially if they claim they are open source.
You can track that in my topics that is updated on regular basis like this one for example: LIST - Open Source Hardware Wallets.

legendary
Activity: 1862
Merit: 1114
WalletScrutiny.com
Hi dkbit98,

WalletScrutiny is a ton of work and we are a small team, only.

In our Methodology you can read our priorities:

Quote
1. Re-evaluate new releases of Reproducible   wallets as they become available. If users opt for a wallet because it is reproducible, they should be waiting for this re-evaluation before updating.

Today I tested the latest releases of AirGap Vault and Green Wallet. Today, Green was a bit more work than usual.

Quote
2. Check if any of the Unreproducible!   wallets updated their issues on their repositories.

We really hope to see more reproducible products, so we always have an eye on the dozens of open issues.

Quote
3. Make general improvements of the platform

That is the a catch-all for improving scripts, design and often just investigations. It's probably the bulk of the work.

Quote
4. Evaluate the most relevant Development   wallets

For Android we have a good proxy for relevance - downloads. For iPhone we don't and neither for hardware wallets.

Unfortunately we are not progressing in the top category as fast as I wish we would but that has to do with severe lack of people to work with code. The k.o. criteria (custodial, bad interface, defunct, ...) are verdicts relatively inexperienced Bitcoiners can come to but when it comes to reproducing a wallet, it's mostly on me. Emanuel also does play with code and does a ton of work but refuses to open merge requests, so writing the difficult reviews is all on one person that also looks into all the other stuff.

So ... if you want to help, there is a ton to do from simple triage to compilation to design to spreading the word. Wink
legendary
Activity: 2212
Merit: 7064
WalletScrutiny website added many hardware wallets on their website and only four of them had reproducible codes, Trezor model One, Trezor model T, BitBox02 and KeepKey.
I was a bit surprised to see that ColdCard wallet is still under development, but maybe @giszmo and his team didn't have enough time to finish testing for ColdCard and other wallets that known to be Open Source.
They made several categories like Defunct (feature many dead wallets), No Source (Ledger), Bad Interface (Coldlar, Secalot, Bepal), Leaks Keys (Opendime), Development (ColdCard and many other wallets), and No BTC category.
I noticed some hardware wallets are missing from the list, like Keystone that should be open source, and it is now replacing defunct Cobo hardware wallet.
Clicking on each wallet is showing small window with basic information, price, size, review date, links and detailed full analysis report.


https://walletscrutiny.com/?verdict=all&platform=hardware

I have the give props to giszmo and his team for keeping their promise and doing this huge work of adding hardware wallets like they promised.
legendary
Activity: 1862
Merit: 1114
WalletScrutiny.com
If you think like that than you should not consider Lightning Network Bitcoin as a real Bitcoin and any wallet that is using LN (custodial or not) should not be on WalletScrutiny website.
Bitcoin Blue wallet is not custodial, and you can create separate page for all Lightning Network wallets and other second layer solutions if you want.
LN Blue wallet wallet can be custodial and non-custodial and there are many shitcoins that can work with LN and not just Bitcoin.
Just my suggestion.

LN Blue wallet is by default custodial and does not warn the user.

I see your point for LN-only wallets like Phoenix but else, the protocol not being as good as Bitcoin in the presence of an actual non-custodial Bitcoin account doesn't make the wallet custodial. Maybe Phoenix is "not a BTC wallet" but certainly not custodial.
legendary
Activity: 2212
Merit: 7064
...

If you think like that than you should not consider Lightning Network Bitcoin as a real Bitcoin and any wallet that is using LN (custodial or not) should not be on WalletScrutiny website.
Bitcoin Blue wallet is not custodial, and you can create separate page for all Lightning Network wallets and other second layer solutions if you want.
LN Blue wallet wallet can be custodial and non-custodial and there are many shitcoins that can work with LN and not just Bitcoin.
Just my suggestion.
legendary
Activity: 1862
Merit: 1114
WalletScrutiny.com
So again, please show me one wrong categorization!

You can run your own node with Blue wallet or you can use their hosted Lightning wallet like for most LN wallets.

Sure, you can but nothing tells the user he should and the website and wallet description claim self-custody while the default LN account is not self-custodial and "- This wallet is
hosted by BlueWallet." does not convey the fact that they can do whatever with the user's funds.

Are you considering Green wallet by Blockstream with Liquid Network custodial or not?
Because I see it is very high rated on your website, or you think Lightning Network Bitcoin and Liquid Network Bitcoin L-BTC are not equal with real Bitcoin.
Looks like a double standards to me, but maybe I am wrong idk.

I personally would not touch Liquid Bitcoins as the current setup is not self-custodial to my own standards but I do not dig deep into all the shitcoins and protocols and personally draw the line around BTC. So if 8(?) federation members collude, they can steal your coins? There are bugs where the federation collapses and Blockstream can single-handedly spend the bitcoins? Yes, not something I would want to get tangled up with but it's not deceptive on the wallet level. It's only deceptive on the protocol level. The wallet does nothing wrong. If I would categorize it as custodial, I would have to do the same with all that support any shitcoin.

Please read the verdict explanation on all the non-verifiable wallets including the custodial ones:

Quote
The app cannot be independently verified. If the provider puts your funds at risk on purpose or by accident, you will probably not know about the issue before people start losing money. If the provider is more criminally inclined he might have collected all the backups of all the wallets, ready to be emptied at the press of a button. The app might have a formidable track record but out of distress or change in management turns out to be evil from some point on, with nobody outside ever knowing before it is too late.

WalletScrutiny is about providers of binaries, currently on the Play Store and the App Store, not about the protocol maintainers/developers.
legendary
Activity: 2212
Merit: 7064
So again, please show me one wrong categorization!

You can run your own node with Blue wallet or you can use their hosted Lightning wallet like for most LN wallets.

Are you considering Green wallet by Blockstream with Liquid Network custodial or not?
Because I see it is very high rated on your website, or you think Lightning Network Bitcoin and Liquid Network Bitcoin L-BTC are not equal with real Bitcoin.
Looks like a double standards to me, but maybe I am wrong idk.
legendary
Activity: 1862
Merit: 1114
WalletScrutiny.com
Interesting to see that no wallet has ever been audited and only few of them are reproducible, but I doubt if any information from this website is really accurate and I don't see any hardware wallet listed.
You have Bluewallet listed as Custodial, and it is clear that this is non-custodial open source wallet, and there is no provider that holds the coins.
This is probably one of the best Bitcoin mobile wallets today.



github:
https://github.com/bluewallet/bluewallet

The "audited" section is to avoid confusion of what we do. We do check reproducibility. That is we test if reviewing the code has any relevance for the binary the provider released. We do not audit wallets. Others might have audited wallets and certainly wallet providers make that claim.

If you find any factual errors, please let us know, ideally via our gitlab. The verdicts are very objective and follow the "methodology" linked in the top of the site.

We are exploring what to do about hardware wallets. Those work very differently and need a very different methodology. We will first expand to other software wallets.

The fact that you thought BlueWallet was self-custodial while implying to know the product tells me everything about why we have to keep the verdict as is for the time being. The provider added a pathetic "This wallet is hosted by BlueWallet" in the LN account creation and calls that a disclaimer.

So again, please show me one wrong categorization!
legendary
Activity: 2212
Merit: 7064
Interesting to see that no wallet has ever been audited and only few of them are reproducible, but I doubt if any information from this website is really accurate and I don't see any hardware wallet listed.
You have Bluewallet listed as Custodial, and it is clear that this is non-custodial open source wallet, and there is no provider that holds the coins.
This is probably one of the best Bitcoin mobile wallets today.



github:
https://github.com/bluewallet/bluewallet
legendary
Activity: 1862
Merit: 1114
WalletScrutiny.com
the right way of implementing a multi signature scheme as some sort of 2FA is how Electrum does it meaning a 2of3 set up where the user owns 2 keys and the third party server owns the one key. user stores one of his keys in his hot wallet and the other he backs up by writing it down on a piece of paper. then if some day the server had any issues he can easily access his funds by accessing that backup key.
the github link suggests that greenwallet supports this but apparently not by default?

That is a good point. As they have to get the user to do a backup anyway, pushing to make two separate backups should not be that awkward and it would solve the problem with the timelock being a timelock when you might need the money.

Ping me on this issue if I forget to update the article.
legendary
Activity: 3472
Merit: 10611
I might be wrong there but my understanding is that the script is a slight bit more complicated. Their 2of2 protects you as you can define rules and they enforce them by not signing if somebody tries to empty your account all at once but if they disappear or charge a huge fee, your funds can be spent with just one key - your key - after one year.

The use of "non-custodial" is completely wrong. Perhaps we can describe them as "Split Custody Wallets."
The issue is gray, you can spend coins even if the network is not available, but you will need to wait & some effort. I think beginners should be warned about this.

(2of2 Recovery Case)
You can spend using nLockTime feature, which enables you to sign transactions by default after a certain time "90 days by default"  then use a tool to be able to send your coins.

Read more ----> https://github.com/greenaddress/garecovery

the right way of implementing a multi signature scheme as some sort of 2FA is how Electrum does it meaning a 2of3 set up where the user owns 2 keys and the third party server owns the one key. user stores one of his keys in his hot wallet and the other he backs up by writing it down on a piece of paper. then if some day the server had any issues he can easily access his funds by accessing that backup key.
the github link suggests that greenwallet supports this but apparently not by default?
legendary
Activity: 1862
Merit: 1114
WalletScrutiny.com
The use of "non-custodial" is completely wrong. Perhaps we can describe them as "Split Custody Wallets."
The issue is gray, you can spend coins even if the network is not available, but you will need to wait & some effort. I think beginners should be warned about this.

(2of2 Recovery Case)
You can spend using nLockTime feature, which enables you to sign transactions by default after a certain time "90 days by default"  then use a tool to be able to send your coins.

Read more ----> https://github.com/greenaddress/garecovery

I see your point and this is not the only wallet where things are not as black or white as we would hope for. I personally consider it a great and unique feature with little down-side but I would also love to allow critical voices to be accessible from the project. What about a block with a Twitter feed showing tweets mentioning both the wallet and @WalletScrutiny? Would also help to spread the word.
legendary
Activity: 2688
Merit: 3983
I might be wrong there but my understanding is that the script is a slight bit more complicated. Their 2of2 protects you as you can define rules and they enforce them by not signing if somebody tries to empty your account all at once but if they disappear or charge a huge fee, your funds can be spent with just one key - your key - after one year.

The use of "non-custodial" is completely wrong. Perhaps we can describe them as "Split Custody Wallets."
The issue is gray, you can spend coins even if the network is not available, but you will need to wait & some effort. I think beginners should be warned about this.

(2of2 Recovery Case)
You can spend using nLockTime feature, which enables you to sign transactions by default after a certain time "90 days by default"  then use a tool to be able to send your coins.

Read more ----> https://github.com/greenaddress/garecovery
legendary
Activity: 1862
Merit: 1114
WalletScrutiny.com
Once they claim " non-custodial," this does not mean that they are telling the truth.
The company uses multi-sig addresses, meaning that in some cases (2 of 2 address) there are two private keys for sending currencies, the first is yours and the second is for the company.
Indeed, the company can not spend money without your permission, but you can't.

I might be wrong there but my understanding is that the script is a slight bit more complicated. Their 2of2 protects you as you can define rules and they enforce them by not signing if somebody tries to empty your account all at once but if they disappear or charge a huge fee, your funds can be spent with just one key - your key - after one year.
legendary
Activity: 2688
Merit: 3983
Stop recommending Blockstream Green Wallet. They are baddddddddd. A big one
sorry about that. I missed clicking on send.
I edited it.

That said, what is so bad about Green Wallet?

Once they claim " non-custodial," this does not mean that they are telling the truth.
The company uses multi-sig addresses, meaning that in some cases (2 of 2 address) there are two private keys for sending currencies, the first is yours and the second is for the company.
Indeed, the company can not spend money without your permission, but you can't.


What will happen if the internet crashes in the area where the company is located, you will not be able to spend your money. The same thing happens when they charge high fees. Also, you cannot claim Hardforks.

Therefore, I do not recommend using it for beginners, or at least tell them about using 2 of 3 addresses.
legendary
Activity: 1862
Merit: 1114
WalletScrutiny.com
Stop recommending Blockstream Green Wallet. They are baddddddddd. A big one

We are not recommending any wallets. Our hope was to drive awareness for the issue of verifiability and there is bad things to say about all 3 wallets listed as "verifiable" but no wallet is perfect and all the other wallets are potentially losing all the money of all their users at once without security researchers having a chance of detecting it before it happens. And most likely even the team is not exercising build verification, so a release manager in distress might be all it takes for all users losing their money.

That said, what is so bad about Green Wallet?
legendary
Activity: 1862
Merit: 1114
WalletScrutiny.com
On your website, you said that Trust Wallet has no source ("Without public source available, this app cannot be verified!"). But they do have a GitHub https://github.com/trustwallet. Does this mean you don't take that as a source or you can't find the repo for the app? Or this is because of that appid thing?

They do have a building guide tho https://developer.trustwallet.com/wallet-core/developing-the-library/building, with the source to be https://github.com/trustwallet/wallet-core. Did anyone try it yet?

Please read the article on that wallet. It explains all we did to come to our conclusion. Let me know if that finding is outdated.
legendary
Activity: 2170
Merit: 1789
On your website, you said that Trust Wallet has no source ("Without public source available, this app cannot be verified!"). But they do have a GitHub https://github.com/trustwallet. Does this mean you don't take that as a source or you can't find the repo for the app? Or this is because of that appid thing?

They do have a building guide tho https://developer.trustwallet.com/wallet-core/developing-the-library/building, with the source to be https://github.com/trustwallet/wallet-core. Did anyone try it yet?
legendary
Activity: 1862
Merit: 1114
WalletScrutiny.com
At WalletScrutiny today we finished our first round assessing the 84 apps we had found to look like maybe being relevant Android Bitcoin wallets. The results are grim:

  • 3 are verifiably built from the project's published source code
  • 21 apps claim to be open source but either we failed to compile them from the information provided on their repositories or the compilation result differed non-trivially from the app found on Google Play. Trivial differences would be file timestamps, differences in few files that can be quickly understood to be harmless, like an API key not being included in the repository, although that is pointless as it sticks out in the diff even more.
  • 25 apps are closed source meaning neither the Playstore description, nor their website nor GitHub searched for their appId revealed any source code
  • 19 apps are for custodial services, the biggest being Coinbase. Coinbase recently reached 10 million downloads and with no other app reviewed having even 5 million, that is more users on Coinbase than on all open source wallets combined. Being your own bank ... not so much Sad
  • 18 apps turned out to be either not wallets, not for Bitcoin or they had only 1000 downloads or less.

This project is only getting started. If you want to look behind the curtain and maybe want to contribute, source for the website is public.

Now the next steps are:

  • Automate verification for wallets that were verifiable once
  • Efficiently collect wallet updates
  • Alert when verification fails
  • Build awareness

If you don't understand what this is about or think it is not important, consider this:

If you are the release manager of a wallet, would you tell your brother to trust your app? Should you trust it? After all it was you who pushed that compile button, right?
Well, if your computer has a backdoor, your compiler might bake in some wallet-stealing "feature" into every version of your app without your knowledge.
How big is the incentive to plant such a backdoor? For some wallets it is gigantic. Hundreds of millions of dollars. Criminals would kill for that amount, which brings me to the second issue:
What if somebody puts you under duress? If whatever you build is not being verified by a second person, ideally far away on an unrelated system, you can't trust yourself and nobody can trust you to release the software you should release. If in an open source project, verification is not easily possible, most likely it is not done internally.
Jump to: