Topic: Why is Armory sending our *USERNAMES* to bitcoinarmory.com ‼️ (Read 9248 times)

Just posted 0.92.2-testing.  Internal testing shows it is good.   Use --tor or settings menu options to disable all network-related actions other than P2P messages to the local Bitcoin Core instance.

https://bitcointalksearch.org/topic/m.8545742

Also updated the offline bundles for 12.04.5 and a significant performance/stability improvement for OSX (thanks doug_armory)

App Nap has been disabled for people running OS X 10.9+. This will result in much-improved performance. In addition, there is a Qt patch I created that I'm in the process of trying to get approved/modified. This patch is needed to prevent some crashes that were still occurring in 0.92.1. I'm still troubleshooting some minor OS X bugs but at least I haven't seen any more crashes since creating the Qt patch.

I'm looking forward to 0.92.2 since I am waiting to package that version.

I had taken a look at the code diff on Github and did not notice any bugs. I also ran the privacy fix branch of Armory, but didn't do much (just ran it and clicked around some of the menus).

Okay, so we've had zero reaction to the latest code, either running it or in code review.  I will try to post a signed build of it shortly, but I was hoping to have one of the original reporters of the issue to review the result (for which I posted a link to the diff on github in a previous message).  We will do some internal testing, but at the moment I can't officially release it in its current state.  I'll see what we can do this week to get it out there.

P.S. - There is a 0.05 bounty per bug!   You have every reason to try it out!

P.P.S. - We found a subtanstial improvement for OSX performance that was a small change so we decided that should go into 0.92.2.  Also should have minimal impact (simply disabling AppNap, I think). 

All the changes are to Python files so, again, no need to compile anything.  Python is interpreted so you simply save your changes and run.  It's also really easy to debug by adding print statements in the code so it'll output to the console exactly what it's doing.  For example, if you wanted to see the URL it's using, add a print statement in the getDecoratedURL function right above the return statement like:


That way it prints to the console exactly what the function returns to the caller.

etotheipi, it's great that you taking action so quick. At first, you were digging yourself a bigger hole by what seemed like excuses. I was about to jump ship to another wallet, but glad to stay now.

Actually, I'm just happy it finally works on XP. However, it seems it wants me to "install" bitcoin ... I'll figure it out, don't worry, that's just a problem on my own computer. (It's set up in such a way that all apps don't know about any other apps, mostly.)

Glad the fud is over. Let the guys go back to bip32 and armory integration.

That's awesome, great work!

I believe on Ubuntu running it through terminal you can see the fetches/etc and all that comes along with it. If anyone would compile it and run it that way it should be easy to spot if it's still repeating previous behavior.

Sadly myself I'm not familiar enough with code to review it.

Thanks again and awesome work!

Good news!  I have implemented all the changes I mentioned earlier today.  It's all in the privacyfix082014 branch.   Here's the full diff between the current master and the new code:

https://github.com/etotheipi/BitcoinArmory/compare/master...privacyfix082014

There are no promises that this is totally complete and bug-free.  But I tested all the different settings, modifying the settings file to make it look like it's a new month, etc.  So far, it all looks good.  But admittedly I was a little rushed.

The bad news is that I am preoccupied until Friday (which is why I rushed), so I won't be able to do any formal testing or final releases before then.  But I wanted to get this out ASAP and get people reviewing the code, as well as let anyone test it that knows how to checkout the repo.

I will start the testing by offering 0.05 BTC for bugs found in that branch.  This only applies to bugs related to privacy settings, announcement checking, etc.  Although we would like to know about unrelated bugs in general, for now we are trying to stay focused on getting this bug fixed ASAP.    To 1a5f9842524:  I believe you said you ran some network tools to find the original issue.  Can you do this again to make sure I didn't miss something? 

Thanks! That sounds great. Thank you for your continued development.

Absolutely great! Thank you so much - this is very reassuring. This solution seems to solve everything that I was worried about.

For a while I thought maybe Armory (the company) had been served with some scary paperwork and told to collect info from users machines. Obviously I was wrong, clearly you guys care about privacy (and security of course) and you listen to your users (something a lot of software devs don't do). Again thanks for doing this, I'll be sending a donation soon and another once 0.92.2 is released and I take a look at the changes.

Armory FTW.

+1

Okay, so here's my plan.

I'd like to have a new version out by next week (0.92.2).  Given that the change is small and doesn't go by any critical code paths, the release testing process can be relaxed slightly.   We'll try to have it out by Monday.

Second, the way I'd like to do it is to put a 4-byte random identifier in the settings file, and use that to for duplicate detection.  That identifier will be overwritten/changed every month, so that any thing that would care about trying to match IDs to systems will expire after a month.  This allows us to aggregate up to monthly statistics.  Anything longer than that we can do without.  

Third, we will decouple the announcement stuff entirely from OS/version reporting.  We will add an option in File->Settings to completely disable this.  Then announcement fetching will send a bare string with no extra metadata.   And as suggested, no extra data needs to be sent on subsequent announcement fetches.

Fourth, we will add a command-line option called "--tor" (with an equivalent option in the settings).  Then we will adapt the code to use that flag  to implement all the standard Tor-based settings:  most likely "--skip-announce-check --skip-online-check --satoshi-port=X".  You guys will be able to examine what code paths are affected by this setting and make recommendations for us to improve it.   (note there is also a "--skip-version-check" flag, but that is no longer used, since we updated the announcement system in 0.91).

I want to reiterate that we do care about privacy -- I've personally been a proponent of security and privacy on these forums for years.  And it would be tough to see why we would really care to match up IDs with announcement fetch pings.  We're not in the data collection business, no advertising, nothing.  We wouldn't know what what to do with it even if we saved it.   We (I) simply made a judgment error when implementing this (compounded by the fact that it was developed as part of a feature we wanted to aggressively promote -- security announcements).  Armory is a massive program, and it is respected for being thorough and careful, but we can't get 100% right.  One of the nice things about open-source is that people can find issues, call them out, and get it fixed.  And that's exactly what happened here.  Thanks for your guys' patience and we'll get this fix out there.

  

Problem is, I love Armory.

Simple don't use armory, if you don't like it. This is absolutely unacceptable. This should be fixed right away.

Awesome.  I will start using this if there isn't an official fix within a week.

Well on the bright side Armory is open source so I forked it on GitHub to remove this behavior.  It looks to be done entirely in a single Python file so you don't even need to rebuild anything to make the change.  Just merge these changes into your announcefetch.py and you should be good to go.

https://github.com/weirddan455/BitcoinArmory/commit/09207dc089facfcea279e485077ed0848491995b

A very nice, professional response.  I'm glad to see this issue is being addressed in a transparent manner.

This is definitely concerning to me.
Thanks for blowing the whistle and thanks for accepting the criticism, devs.  Thumbs up all-around here.

It seems like a GUID would be a great fit for this requirement. Try to load it from the settings file, if it's not there generate one and save it.

Well, if you have a properly configured router or proxy, Armory will never get to send any data out.

The best way is to accept the user's honesty and request for this information upon download. Meaning, just count the number of downloads from the armory website.

It's not perfect as some people will download the next version more than once.

You can't filter out dupes this way, as some people might use proxies or VPNs or coincidentally the same TOR exit node. And you'll miss all the other numbers from people mirroring the site.

I download the install files once from a different computer. I install it on my other computer or in a VM. I also use Deep Freeze, so often times Armory has no memory that I've already installed it yesterday, because as far as it's concerned, it's being installed for the first time.

Blockchain constantly tracks you

Or just don't use an identifier at all and only send the stats once and keep track of whether they have been sent or not somewhere in the data directory. In the case of an update include the last version that it was updated from so that you guys can adjust your stats.

There is very little reason to need to have a unique identifer for each install. No other wallet does this that I know of.

Here's a suggestion: randomly generate a 32-bit unique identifier, store it in the Armory config files, and report that. It's not perfect, since someone with multiple folders will be counted multiple times, but it's pretty good.

You're misunderstanding that. Try to configure Armory to use a proxy: you can't.

Ditto

Once again we have just witnessed the true beauty and power of open source. Thank you 1a5f9842524 for ringing the alarm, and thank you etotheipi for fixing it.

i smell poo.. OH SHIT.. its on my shoe.  just stepped in some armory poo and its too late

Armory will always be the best.

Suggestion: Generate a completely random token upon first run. Store this token in the armory data directory, and optionally remove it upon uninstall.
Bonus points: Make it opt-out during install.
Additionally: Destroy all previously collected hashes. To retain usage data just replace them with unique random values, or an incrementing counter.

I'm really glad this was found and the response from the Armory dev was very reassuring after my inital panic. Thank you for all your hard work etotheipi!

It isn't that Amory doesn't use the configured proxy rather that Armory doesn't have the ability to configure one as the only connection it makes to the internet are these pings (as far as I know).

Armory uses bitcoind as a backend. Bitcoind connects to the bitcoin network via the proxy it is configured to connect to. This is how people use Tor with Armory, you configure bitcoind (not Armory) to use Tor. Most of the people who do this likely have no idea that Armory is connecting to the internet at all nevermind outside of Tor. I assumed that I had to press "Check for updates" for it to dial home. This is obviously bad for many reasons. A lot of people use bitcoind over Tor to hide the fact they are a bitcoin user from their ISP. The ISP can see these regular pings to bitcoinarmory.com every 30 minutes and figure out they are likely a bitcoin user.

It should be possible to disable the announcement pings in the GUI, maybe have a big scary popup appear when the user tries to disable them that lets the user know that they should manually press "check for updates" and periodically check the website for updates.

Thank you.

Thank you for "'fessing up" and good choice to separate announce and statistics.

But most problematic though is (as was reported above) that these requests are not going through the configured Proxy. If that is really so, then Armory users are quickly and easily identified by their ISP, which can be very problematic in some countries.

Sorry guys, I've been out all day, but I've been thinking about this.  You all are absolutely right.  We made two mistakes:

(1) We assumed that because we choose not to store/process the IP data, that users would believe us that we don't
(2) We incorrectly assumed the that space of user home paths on your desktop was big enough that the 4 byte identifier would not have adverse privacy implications. I did not consider that people's home usernames might be, say, their username on bitcointalk.org

It's quite clear that those two pieces of data do have pretty serious privacy implications.  I want to fix this ASAP.  

To be clear, the reason we made the unique identifiers is because we don't want to store any IP data for the reasons described: if there was a subpoena of some sort, we'd prefer to not have anything to reveal.  And we don't store it.  But, we incorrectly assumed that the unique identifiers would be sufficiently non-privacy-leaking while still allowing us to remove duplicates (in response to justus: we want an identifier that will be persistent between loads so that users that start the program over and over are not counted for each start--as we all know, a lot of users have difficulty with Armory and will start it 300 times to try to get it to work).  It is clear these were bad assumptions since we technically could be storing both which would be quite bad.  

I hope we've generated enough good faith with the community that we get a little slack that this was not our intention.  I take the blame for not realizing that, and I want to make sure that it is fixed.  ASAP.  I will happily take feedback on how this should be adjusted so that we can meet our goals without compromising the privacy of the users.  

I agree we should decouple the option from the announcement fetching.  We consider announcements to be extremely important, and why we made that difficult to disable:  if there's a critical security (or privacy!) vulnerability in Armory, there is a very short window where someone might try to exploit it to steal peoples' coins, and there's no better way to help users than to make sure a big scary warning pops up the next time they start Armory.  The fact that we coupled the OS/version reporting with meant that it was as hard to disable that as it is the announcement fetching.  We can easily separate them and will happily make it easy to disable the OS/version reporting.


Re privacy policy:  On the advice of our lawyer, we included the "may collect IP addresses" because we have no way to prove that we don't.  And since our website uses google-analytics, we don't have control over what google does with the access patterns of users to our website.  It was a bit of CYA that companies have to abide by, especially in the US.  Note we describe at the bottom of that page we describe how to disable it with a link to our troubleshooting page.

On the upside: another positive example of the power of open-source software.  We have casually encouraged users to go through the code base, and we even contacted the Open Crypto Audit Project to try to get a code review (and never heard back from them).  We are obviously believers in open-source, and here's the first solid example of Armory getting better because of it.  We will get this fixed.

I'm not a programmer and certainly wouldn't assume manually modifying a file to be "opting-out."

This is pretty alarming behavior. Guess I won't be using Armory anytime soon.

Measure userbase by number of downloads. Truecrypt did that for 10 years.

A more responsible way to do this would be to generate a random unique identifier, instead of one that could be guess ahead of time, and make phoning home an opt-in feature.

Your claim that user can opt out is disingenuous - most users aren't going to know how to change a .desktop file to alter command line options.

For practical purposes options that can't be configured through the GUI don't exist.

You're of a very small majority.

Not really. One can broadcast the raw tx let's say on blockchain.info when the client is closed. I did it a few times myself.

Sure they can.

You know that the user won't be making transactions when the application is closed, so you can start to eliminate transactions on the chain which couldn't possibly have been theirs. Armory doesn't use compressed point pubkeys, and does not reuse addresses which you can use to further filter the transactions you see. With the last two features alone you can eliminate a large portion of all transactions in each block as being not possibly being made by Armory.

Having 32 bits of the home folder hash is actually pretty devastating to your privacy as well. Within that space you could search bitcointalk user names for example with only a 1 in 4 billion chance of a false positive per attempt. I'd bet on a lot of people having user names that very much match with their bitcointalk ones.

Running armory doesn't necessarily mean that you are performing transactions, so i don't think any connections can be made by that.

In any case it's nice that you have noticed that and i think a permanent opt-out option should be available.
Then again stats might be one of the few ways for creators to monetize their wallet-software.

Your Privacy Policy (which oddly enough I've never seen before now) says that you only log IP addresses. Which is it? 

https://i.imgur.com/6xPsGpU.jpg

Collecting 32 bits worth of people's usernames is not mentioned in the Privacy Policy at all, ergo you shouldn't be doing it. It also mentions nothing about sending it to Amazon S3, or to Cloudflare which you use to host the ping domains.

I agree.

On the website, they have a privacy policy that states this:


However I don't recall being forced to agree to this when I ran the software, though from looking at the source it appears to be in the help menu at least.

Why don't you use the alert system in Bitcoind just like everybody else? It would be instant and more reliable, only you wouldn't be able to collect personal information with that method.





Utter nonsense.

If you wanted a unique anonymous ID you would have generated a few random bytes and used that. Instead you used a highly identifying, personal piece of information and sent it to your remote server along with the IP address of the user. There's no way you can pretend that was a mistake from somebody who is writing wallet software.

Why don't you do us a favor and delete all the information you've collected without your users consent.

I still don't know why you need to know their installation ID for each ping. Can you come up with a real-life example of a situation where you'd need to know that information when providing announcements? if there were critical issues in Armory that would affect everyone running that version and not specific installations.

For most users privacy is just as important as security.

Annoucements/statistics should be two completely seperate things. I think statistics should happen on first-run ONLY and it should be very obvious it's happening and easy for the user to opt-out. Checking for annoucements should send the bare amount of information you require, the Armory version and platform.

I was under the impression checks for announcements weren't automatic and it seems others were too. Maybe it should be more obvious that they are automatic and easier to disable them (in the GUI) for people who want to check manually instead.

So that we don't "count" that ping as a unique user.  Our goal is to get a rough gauge of how many people are using Armory, and what the OS & version distribution is.  That's all we use the data for.  If we send a ping without the ID, we don't know if it's a duplicate. 

Also, I shouldn't have suggested "just" hard-forks... a piece of secure software used by people with massive amounts of money has many different reasons users might need to be notified, including critical security issues with Armory, if they arise.

You don't need to send the installation ID when checking for announcements. Why would you need that?

The 30 minutes isn't to for "collection", it's for announcement checking.  If there's a hard fork and people are potentially going to lose money, we need users to be aware as soon as possible.

Additionally you could still get a list of all usernames on bitcointalk for example and compute the ID hashes by working out what their home directory would be expected to be if they used the same username on bitcointalk and their PC.

You'd be able to check that hash against your "statistics database" and find their installation along with all of the IP's that gave you the statistics.

What you do is arguably worse than simply collecting IP's.

Why don't you just send the OS version etc without the ID on the first time Armory is run? Much better than what you do.

There should be an option on install to disable this activity of the Armory client.

That's pretty much synonymous, most people will have their user name set to either their common pseudonym or their full name. If you have a batch of people who you think might have sent a transaction, 4 bytes of the hash is more than enough to work out which one.

Sending any personal information at all is "not cool", especially when nobody was told about it in the first place.



Why ping every 30 minutes?



You can't expect a user to apply this every single time they load up the client. Not giving the option in the GUI is intentionally obstructive.

If that is the case, why do you need to collect statistics every 30 minutes? By doing that you also see what times the installation is running which could allow you to match it up against bitcoin transactions made during those times.

And whether you like it or not the IP is transmitted along with the unique installation ID and we can't know whether you are storing the IP's or not. This bypasses any proxy settings set on bitcoind, such as people using Tor.

Guys, calm down.  

The code you posted doesn't send your username to bitcoinarmory.com, it sends the truncated hash of your user home directory path.  This does not give us any information about you except that it will be the same when your system makes multiple requests for version/announcement information.   We intentionally chose this instead of tracking by IP because we knew that IP logging was "not cool".  And in the end, we don't care about your IP, we only use it the ID for collecting statistics about what operatings systems are being use to run Armory and what versions people are using, especially after announcing new versions.  This helps us remove duplicates.

Armory (the company) only tracks unique IDs long enough to collect daily statistics of our user base, like how many people have upgraded.  If a announce-request is made and comes from an ID we have never seen, we add the OS and Armory version to the statistics.  Otherwise we ignore it.   That's it.  We added the unique ID so that we have a way to count unique users without logging IP addresses.    We also add the ability for you disable this by running with "--skip-annuonce-check".  

As a company, we have to have some way to measure our userbase, and we felt this was the least intrusive way possible.  And you can opt-out.

Why is this code in Armory?  What purpose does it serve an end user of Armory?

Here is where USER_HOME_DIR is set:
https://github.com/etotheipi/BitcoinArmory/blob/master/armoryengine/ArmoryUtils.py#L265

On my Windows machine it gets set to this: C:\Users\\AppData\Roaming

It seems to send the first 4 characters of the hash, enough information to distinguish between different Armory installations.

Wow, Armory devs, maybe you can tell us what is going on?

This is something to consider, but I have not been using Armory because of two reasons:

1. Doesn't work on 32 bit XP.
2. Doesn't work with compressed keys.

I guess you could just block bitcoinarmory.com on your router or firewall.

I thought that too. All the better to ruin your privacy with I suppose.



How would they know if the requests are duplicate if they weren't logging them?

Holy hell. Thats pretty serious if you ask me. Has anyone brought this up before?

It seems to be for getting the announcements on new releases. Why do you need the hash of the home directory for that?

The GUI gave me the impression that I had to press "Check for updates" before it would "dial home", apparently I was wrong. Why would you do that every 30 minutes? It shouldn't do that automatically!

I didn't see the comment where they admit its logged. But if it is why in the hell would they do that? that puts them at unnecessary legal risk of receiving subpoenas for this information.

Update:
The Armory developers are releasing an update that will fix everything we were worried about, see here: https://bitcointalksearch.org/topic/m.8299712

The Armory client makes a HTTP request bitcoinarmory.com every 30 minutes with the hash of your home folder name, OS version, and of course your IP address (proxies are ignore) at every start of the application. The only way to disable it is not an option that can be set, but a command line option you need to use every launch. The options "skip online check" and "disable software upgrade notifications" do not disable it as one would expect.

It's not hard to imagine how this would be connected with the transactions a person makes just due to the timings of the requests. Even just matching the hashes returned with usernames on this forum would be enough to identify a lot of people. As pointed out below, this does not respect proxy settings so the real IP of the user is sent to bitcoinarmory.com.

• How much of this is being logged by Amazon S3 and bitcoinarmory.com?
• How long are the logs kept?
• Why aren't users told their privacy is being violated?
• Why can't this feature be turned off in a sensible way?

This behavior is totally unacceptable.


https://github.com/etotheipi/BitcoinArmory/blob/7bd89850a90a280e3345c29d7e0338b62f841548/announcefetch.py#L232