Topic: debian 11/bullseye deps (Read 204 times)

bullseye has 10.2, probably will get 10.3 in the end, if we're lucky



nope, gcc 10.2 won't eat it.

This line? Initializer list should build. I'm guessing your version of gcc is older than mine (11.1). Try this maybe:


This is tested here:

https://github.com/goatpig/BitcoinArmory/blob/dev/cppForSwig/gtest/UtilsTests.cpp#L6545

You may want to reduce the search space, this test can be cpu hungry. ("./configure --enable-tests" to build tests)

lol, and there I was thinking to myself: "seems a little strange for this small python script to recompile this small C interface wrapper every time ArmoryQt.py runs, must mention this design oversight..."


I forgot to mention also, I did need to modify 'cppForSwig/TxHashFilters.h', although perhaps you already noticed this? There's a call to a non-existant assignment constructor for struct TxHashHints on ln 560 of 'TxHashFilters.cpp', so I wrote the constructor in the header file.

No idea what kind of parameter is best, as there's too much code for me to take in so fast  (but it seems ok, as the object is move constructed to a vector a few line later in the same scope). So I passed by value. If this causes the end of the world, I can only apologize.

1. Submodules aren't obvious. Subtrees are a bit better. Submodules need to be setup and updated manually. There is very little indicating that a submodule even exists in a project, let alone that you have to update it. I used submodules before, I've had veterans trip up on it. I have tripped up on it myself. No amount of git pull or git reset in the root folder will checkout the right submodule revision.

2. If you submodule a repo, you have to integrate its build process to yours. This quickly becomes a nightmare if you are using divergent build tools across the project and submodules (autohell vs cmake vs ninja vs whatever else).

3. If you need to amend the code in the helper lib, submodules are straight up useless. You'll need to fork the repo and commit your changes. Then you'll need to keep your fork up to date with upstream, and submodule your fork.

4. If you dont need to modify the helper code, you essentially have no reason to keep a local copy. Linux is designed around system libs, so there is very little extra effort there to use helper libs. On Windows the submodules provide low value, since the lib you're using on *nix may simply not exist for Windows (case in point, GMP vs MPIR).

5. To build against a helper lib, you need the headers and the binary. This means you can inject that at build time and manage the helpers whichever way you want. Copying the code locally removes that option entirely. This will more often than not degrade portability on obscure Linux distros.

Won't a git submodule do the equivalent job of a helper script but at a fraction of the effort?

Oh sorry, forgot about that one. Don't modify any code, you simply need to build the python module. In the c20p1305_cffi folder, run


This will build the module. The code will find it on its own after that. This is a one time thing, you won't need to rerun that step unless I update that code.

aha, ok. that all sounds quite positive, then

will probably be back with a question or two, or just to report some initial impressions of Armory 0.96.99

---

so, python runtime errors won't let me start ArmoryQt.py:


if I tell armoryengine/BIP15x.py what python wants to hear (by changing 'import c20p1305' to 'import c20p1305_cffi', it has a different complaint:


I prefer automake errors tbh

[moderator's note: consecutive posts merged]

Ah, that won't work. This arg has the configure script expand the custom path with /libs for ld (to find the binaries) and /include for gcc (to find the headers). It fully expects a cloned repo structure, not a system install (where headers and libs are in completely different folders).


It shouldn't diverge that much.


Until recently, Ubuntu didn't package LWS 3+. I started using LWS when Ubuntu packaged LWS1, and LWS2 just had come out with a new model for the main callback. The 2 versions were not compatible but I wasn't going to wait for Ubuntu to catch up. LWS itself kept pushing major releases in quick succession, with minor backwards compatibility breakage (hence the update to LWS3-4, which don't work with LWS2, but work with one another), and now modern distros carry at least LWS3, so it makes sense to finally allow building vs system installs.


There may be some confusion here. When I started working on Armory, the major open source dependencies were copied as is in the armory source repo (besides python). I guess that's what you're referring to when you say "in tree". I'm trying to get rid that practice. At the moment, gtest, libbtc and chacha20poly1305 still exist as copies in repo. LWS, LMDB and protobuf are out. There is no python dependency anymore. My intent is to evict as much foreign code as possible from the repo before the next release. As a result, at least libbtc and gtest will go too, soon.

Libs were copied in the armory source primarily as a way to ease the building across environments. System libs may vary across distros, Mac has mostly been a blackbox and you can't really install system libs on Windows.

The new approach is to get rid of all the copied code, let the default config look for system libs and allow users to feed it adhoc paths to their own builds of the dependencies. This is how LWS is treated and how all libs will be treated eventually. In this scenario, there's no burden to keep updating the dependencies code, while it's flexible enough to eventually build on any system.

I can later add a helper script that will download the source from the relevant repos, build the dependencies and feed the paths to the cmake project to ease as the "easy to build" DIY solution that should work across all OS. I'll keep on using autohell and manual setups on my own system.


Cleaned the warnings and got rid of SWIG, that should have speed things up. Also, computers did get faster since =D.

yeah, I tried forcing this option, because simply supplying --with-own-lws=/usr/lib/x86_64-linux-gnu didn't work, configure was trying to use '/usr/lib/x86_64-linux-gnu/libs/libwebsockets.a', which was annoying.

I still managed to trick the configure script, but got left with a sea of 'unresolved dependency' anyway. I slightly forgot what that meant, and assumed debian package was the problem, when that wasn't exactly true.

despite my compile working with the Makefile.am edits, there's a real possibility that it will exhibit different behavior and/or bugs to what you're building, right? I'm thinking this because LWS is linking to libev _not_ libuv (although is one not a fork of the other... I am out of my depth here really)


I'm likely to stick with automake where it's available. Seems to me the hardest to use, but also common, so I need to know how to handle it. Of course, some projects are using cmake (or other alternatives),

---

Is it not the case that going with the distro LWS is a better option anyway? Surely having a custom built in-tree LWS will be harder to maintain, especially when debian stable has a relatively recent release (although we should not expect things to stay like that). On the other hand, Armory builds much faster now, unless I'm making a hilarious mistake (dumb Python gui reduces the burden on the build?)

You can try the cmake project in the cmake_build branch. It runs on system libs only atm. It doesn't have the stuff to build CppBridge yet however, but you can check if armorydb will build correctly at least. To setup the build system, do the following from the source dir:


Then you can call make from the build folder. The binary will show up in there too.

Simplest way around this is as follows:

Clone lws locally. Setup using cmake: you can pick between libevent and libuv at setup time, also I suggest disabling openssl usage if you don't plan on using it. Build it and you're done.
When configuring armory, use the following:

Use this custom arg to point to the root folder where you cloned lws, it should just work. This is what I use when I don't want to battle the system lib. There is also a cmake project in a PR which seems to build just fine with system libs (but doesn't have the custom args to inject lib paths at setup time). I could just fast track that too.


There's no Qt4/py2 left in that branch anymore.

If you were on Ubuntu I guess you could also add the rock-core/qt4 PPA and then apt-get install qt4-x11 (no clue about psutil though, sorry). You probably already know the "ubuntu PPA's won't work on debian" caveat by now though.

I don't recommend building Qt4 from the source because it's a general pain in the ass to do and for me it caused a lot of Make errors.

ok, so did a little more work on it, and I have it done... by some definition of the word


in cppForSwig/Makefile.am:

• cppbridge.a and armorydb.a needed linking to libev
• the LIBUV_STATIC variable needed changing to 'libuv_a.a'
• changed PROTOBUF_STATIC variable to 'libprotobuf.a'

some weird changes.

1. LWS must be using libev, whereas the build system expects libuv
2. I guess bullseye changed the expected/conventional name for libuv.a? which isn't being used anyway?
3. seeing as LIBPROTOBUF_STATIC is referring to the static library, switching from the .so to the .a actually makes sense, (although ld complains it's 'not portable')


sorry for the noise anyway. I am too tired to actually test Armory now, will try it tomorrow.

I'll look forward to trying.

However, I'm currently still trying to compile on Bullseye. It seems that stock debian libwebsockets (4.0.28) is not supported (yet?)

configure seems to expect both libevent and libuv as deps, which is a little confusing, as I'd expect that LWS cannot use both concurrently.


am I on the right track? if so, which build options should I ask cmake to build LWS with, seems a little pointless trying to guess.

I was not on the right track, using debian packaged libwebsockets appears to be ok

Latest commit in dev should be able to create & sign unsigned transactions. It may choke while converting wallets, checking that next.

right, these corporate distros are bad for a dozen reasons. tbf, either Nix or Guix would solve this specific problem without too much effort, but I'm still a Qubes kid (and there's no Nix/Guix outside of a self-rolled HVM, without all the clever inter-VM glue that makes it e.g. easy to copy un/signed tx data between VMs, so that's swapping a medium-sized problem for a project-sized problem).


sounds like good stuff. Python to me seems better suited to "one-shot" code; some underlying library that is seldom called for a handful of functions, or some family of scripts, or for prototyping something only. It's too slovenly to use for anything demanding or daemon based. "Dumb dialog boxes" pretty much fits that description.


mmm, i hear that qt is becoming more unfriendly to open-source, so maybe better to jump ship sooner rather than later.

Starting Ubuntu 19, py2 was not installed by default anymore. You could however install it, as well as qt4, and all the necessary dependencies for Armory. That accelerated quickly however and the whole thing was dumped unceremoniously in Ubuntu 21. I don't blame them however. I've lost track of that tbh, I use Manjaro these days. I may just bite the bullet and go full Arch and never look back.

Python is now just a runtime for the GUI. The blockchain service and wallet manager/signer are now fully headless C++ processes. The GUI is a dumb interface that implements as little mechanisms as possible. Every minute operation in the GUI is deferred to the wallet manager, from converting scripts to address strings to coin selection. The wallet manager interfaces with the blockchain service to keep track of balance and transaction history.

I stuck to Python cause all of Armory's legacy UI is hardcoded as qt4 dialogs and widgets. I intent to use QML to implement new UI elements going forward, and then maybe dump the Python/Qt dependency altogether and migrate the QML to something more modern like Flutter.

sure, but I'm not sure how interested I really am in discovering exactly how many qt4 dependencies I also need to compile due to debian 11 being too far ahead, especially if you're somewhat closer towards a functional/working 0.97. The security updates on debian 10 will keep coming for while yet.

strange that your timing in completing this python3/qt5 marathon is right around the time that debian has killed them off! I'd bet that ubuntu (and the rest) did it months/years ago, of course

PySide2. It's an officially maintained version of Qt5 for Python, but with a less restrictive license than PyQt5.


It doesn't, I'd have to use PySide6 for Qt6 (there's no PySide 3-4-5, they just went from 2 to 6)


You can build qt4 from source and install locally (look up qt-everywhere). Maybe there's a pyqt4 version on pip that statically links to qt4 and you wouldn't even need to install qt4.

right, I will get going figuring out the details next week some time.

I'm a little more proficient in python (and even c++) these days, so it will at least be less work (for me) with all the testing and reporting

---

you're using pyside instead of qt? that bypasses the qt version somehow? it seems that qt4 doesn't exist on debian bullseye

Got rid of that dependency in the py3 version entirely. It's used to spawn processes and watch over the PIDs for what its worth.


Unlock wallet flow is a bit busted atm, have to fix it. I suspect the offline signing will throw some exceptions along the way. Nothing design related though, most likely syntax issues (Qt4 doesn't translate 100% into PySide2) and missing imports (I've been refactoring the massive qtdialogs file into a collection of smaller files, in part to get rid of the circular imports). I've signed offline transactions on testnet before but went through some more refactoring lately so I expect some cleanup ahead of me. Could be fixed sometimes in the coming week. That aside, the wallet creation dialogs are probably busted, same with comments (doesn't map over to the new wallet format yet), and the migration dialog from old to new wallets is thread bare.

Be warned, the build system isn't super user friendly right now.

seems like this will not work (easily)

so far:

• no qt4
• no psutil

I'm guessing any psutil functions were rolled into python3 "native" libraries, which doesn't help at all
I guess also there are probably more dependency problems, but no qt4 already seems too serious as things stand.


to what extent is dev branch at github.com/goatpig/bitcoinarmory usable for everyday stuff? Or is this where we all realize that Armory is (for now) pinned to Debian 10 and version 0.96.5?

my recent experience with other packages that need a python2 interpreter suggest that other Linux distros don't make this any easier