Topic: Mailing list migration (Read 457 times)

As someone who isn't a direct participant, I like to periodically read the archives.  If only to get a sense of what developers are thinking and what their attitudes are.  The level of demand from casual observers like myself may be more difficult to ascertain, compared to the demand from those actively posting messages.  I feel the transparency of it is quite important, so would very much like to see that aspect preserved.  Plus the mailing list tends to be more conversational than the equivalent discussions on GitHub, focusing on ideas rather than implementation, which makes it more accessible to those of us who aren't coders.  It would be a shame to see it go.

It is not surprising. If you compare the discussion today, with the discussion in the past, then you note it is much different, than it was. And you also note, that people moved to other places. For example, there are IRC channels, there are GitHub discussions, and also note that the winter holidays are coming.

Another thing is that if you want to see new messages, then someone should accept them. Which means, even if there is some traffic, waiting to be published, you can see it a day or two later, because things are moderated.

Not really, and there is another possible outcome: the list could just silently die, without any replacement. This is also possible, and this will not be the end of the world, because participants know their e-mails, and can exchange some messages directly, without any list, and another list can be formed from scratch, when there will be a need for that. Because if there is little or no demand, then it is normal to have little or no supply.

It is not. It is e-mail based, and also moderated list, where it is normal to post something, and have your message rejected, or published two days later. It is not like this forum, where you write a post, and where it is immediately fetched by some bots, and where all statistics are recalculated based on that, and where people can see the first version, even if you edit or remove it.

Not really. I don't think there is anything under the hood, that you cannot see publicly. I think the main idea is to let it die, and create something new, only if there will be any need to do so. And I think it is a good approach, because there are many ways to communicate, and because if someone would want to talk to any participant on the list, the e-mail address is usually public, so if you want to know, what John Doe thinks about that, then you can just ask him directly.

Well, this turns out to be urgent for the holders of these mail listings.
Have they said anything about this matter? Without their feedback, it is also difficult to debate the ideas presented much further.

Much like the mailing list itself, the discussion here about potential migration just kinda... stopped.  There seemed to be a lot of suggestions, but no clear sense of direction.  Do we at least have some sort of leading contender or front-runner yet?  I got the sense that it's a time-sensitive matter, so I would have thought there might be a greater sense of urgency about it.

Some points and just my view on a lot of them and not in order (sorry)

1) Running a wide open mail server is difficult. Running one for just a mailing list is somewhat simpler as if you are keeping the list clean and only allowing approved people to post will cut down on a lot of spam

2) The cost I really don't see as an issue. Although there is a cost, it can be done for under $1000 a year, which while it is not an insignificant sum it's not earth shattering.
2a) I can see a lot of places wanting to give a discount for getting bit a free advertising for supporting BTC

3) If you want to keep with mailman there might be more coding but there are other free and commercial options out there. I know a lot of people will balk from buying a piece of closed source commercial software to do this but time is money and if spending some money now saves time and effort then it's not wrong to do it.

4) Administration would be an issue, BUT I would think there are some people out there that would want to help. I could be 100% wrong on this, but with the number of people supporting BTC I can't see no one helping.

5) Although a royal pain in the ass you are not locked into 1 option. If you go with product A and discover it sucks....you then try B. I keep seeing the attitude of well this sucks but we are sticking with it from a small 2 person business to major 1000+ employee places and it just boggles my mind. Fine it didn't work....move on.


-Dave

It should be possible if it's one bot creating every post (so I don't need to get someone else’s password). But I honestly don't see the devs using it.

Logically I made a minimalist description of what this bot did.
It would only automatically post a message if the sender was on the authorized list. The rest would have to be approved in advance.

Anyway, I think there are several factors that need to be evaluated and several possibilities.

I think GitHub is a better alternative. Because you can make comments, just by sending them via e-mail. So, this functionality is already there, when it comes to GitHub. Of course, it has some drawbacks, for example moderation, that should be applied before some message will be included, and not after that.

In general, when I look at all proposed solutions, I think GitHub is the most promising one, because development is already here, you have all Pull Requests, you have all source code, and all BIPs. Everything is there, except full mailing list support. But of course, it is not 100% bulletproof solution, but only "not that evil as some others".

Well, the closest thing that exists on Bitcointalk, is probably something like Private Messages. But still, you have to log in, so the protocol to use Private Messages is not the same as in e-mails. If you could just send a Private Message via bitcointalk, without logging in, just by filling all needed fields, and without establishing a connection, that could work.

The bot should not write the post automatically. It should be accepted by some moderator first, and then published. This is how mailing lists work, and that functionality should be preserved. Because you don't want a Mallory, that would send some spam to the bot. And you also don't want a Mallory, that would know the whole list of participants.

I understood. Thank you for the explanation. 
In this case, they are public lists, where everyone can read what is said.

Using the forum would no longer use a tool as universal as email. Well, why not create an equivalent service? There must be scripts already made with these characteristics.

Either way, I think the forum could be a good alternative. And maybe someone could create a bot, where you send an email, and the bot then writes the post.
@TryNinja, was it viable?

I honestly thought this day would never come. Linux mailing lists were the top place for high level technical discussions. So many projects that contributed hugely to society came out of there. Why stop it so suddenly andw here will all these projects go now... Sad.

Imagine you have some e-mail address. You can send mails to anyone you want. If there are just two people, then things are easy. Alice can send any message to Bob, and he can reply. The whole conversation history is preserved on their machines.

Then, imagine you want to add more and more people into the conversation between Alice and Bob. There are three people, four people, then five, and then it can grow very large. So, what happens then?

In that case, if there are 1000 participants in the whole conversation, then Alice don't want to send the same e-mail to 1000 recipients. It would take a lot of resources, if every member would need to send every message to everyone else. So, what happens then?

Then, people create some mailing list. Which means, all of them send their replies just to a single e-mail address they agree on, and then, this single server is used to collect all of their messages, and to send them to all participants on the list. Also, it makes moderation easier, because if you have "pure P2P e-mail with no mailing list", then Mallory would know the whole list of participants, and could send them some spam. Then, each participant alone would receive it, and would need to manually handle that spam.

So, in the larger communities, mailing lists are useful to manage some conversations via e-mail, between N people. And also, there is usually some kind of moderation, because if you include more and more people, then you need to filter some messages, to keep the whole conversation on a sufficiently high level.

It can do so. You can write to some person directly, to the mailing list, to both, or to none of them. You, as a sender, can decide, where to send it (you can send e-mails even to yourself-only, if you really want). And then, the recipient can decide, how to accept it. Some people accept only messages from some trusted circle of senders, some others can accept messages from everyone.

Well, each recipient can deal with its own mailbox, as they please. Which means, you can be a member of some mailing list, but you can arbitrarily decide "I don't want to read messages written by John Doe". Or you can decide that "I don't want to read this topic about Ordinals". This is your choice, and you can just apply it to your message rules, and then it will be handled automatically by your mailbox.

Yes, this is also the case. In general, you can just let other people do the moderation, if this is sufficient for you. But you, as a recipient, can accept less messages, if that is your choice.

How does this Mailing List system work?

Does person A send an email to person B, with a copy of the Mailing List email? This email is then evaluated/moderated and approved or not. And that?
Or are all emails sent to the Mailing List, and then managed by moderation?

Indeed, and I think that is partly why the mailing list has been able to be useful for so long.

For a while, the mailing list was greylisted - anyone could send to the list, but people who hadn't posted before would have their mails held in a moderation queue before they were sent. Apparently issues with mailman2 have made the mailing list now a moderated only list, although they still permit a lot of stuff and it's basically just manual greylisting. The key thing is the fact that emails from new people are held while emails from "approved users" go through pretty quickly. This cuts down on the spam.

However any system which requires pre-approving users before they can send anything will discourage many people from posting. With email, you don't know that your email might get moderated before you send it, but anything that requires approval before being able to even send will just result in fewer new people participating.

---

The fact that email is universal is extremely beneficial. New people do post to the mailing list, and sometimes they have good ideas. I strongly prefer a mailing list for these technical discussions since it has a low barrier of entry. Everyone has an email, and everyone knows how to use one. There's no need to make a new account with yet another password. No need to have yet another website that you have to login to in order to read or respond to posts. The barrier of entry is low, and that's what we want so that fresh ideas can come in. Of course we need proactive moderation so that it doesn't become all spam. Mailing lists are perfect for that.

Of course the last bit that makes everything hard is who is going to host it, where is it going to be hosted, and how will it be paid for. Running a mail server is not easy. It's extremely easy to get blacklisted and then the server is useless. The mailing list also has thousands of subscribers and apparently the existing self-hosted mailing list software just is not capable of dealing with it, or have significant other drawbacks. It's not just "a bit of modification", and just because there are developers involved doesn't mean that all software problems are solvable. mailman2 and mailman3 are complex projects with lots of issues, and the people working on Bitcoin aren't working on fixing those software since they're interested in working on Bitcoin. Even if the software were functional, who would host? Who would pay for servers? These are questions that are yet unanswered, and often have unsatisfactory solutions. Even now, it isn't the Linux Foundation that hosts the list. It's OSUOSL, a non-profit that helps open source projects. But they too have limited time and money.

Wow, it was still hosted on LF? I didn't know.

Looks like bitcoin-dev doesn't have many options since running a private MTA is not being considered. But one interesting thing mentioned is a "forum" they might evaluate. I think, among all the other options, this will be the better one if not the best.

We are a living example that bitcointalk.org is still running, not only running but thriving, thanks to our moderator theymos. No doubt they should explore this option; yes, you will lose the "email" functionality, but at least it will live for quite some time. I think theymos should approach them if they are considering this option since they mentioned "What about bitcointalk.org"; we should welcome it.

I hope it's not going to be Google Groups, at least. I feel this Google product is going to die soon.

"Finally, another option is to do literally nothing."

That sounds like theymos, isn't it?

---

I don't think budget or running a mailman server is an issue but the issue is in its management (management of mailman server).

It's already highlighted in the  mail that managing a Mailman server, especially for a large and active mailing list like bitcoin-dev, can be a challenging task there many things that need no full time moderators not part-time moderators or volunteer members can do this job.

Ideally I also agree that te forum would be a very good place for devs to call "home" to the "dev'ing work. Also it could also be seen as an homage to Satoshi that founded the forum and created the most perfect piece of technology that mankind has seen so far!

The problem seems not to be only spam.
Also the problem of user aproval for posts. As we know, Bitcoin is an open source software and anyone can be a dev. Today you're a dev, tomorrow you're not. And I agree that it woul be quite an effort to keep things working perfectly. I don't know how it was managed in the mailing lists but apparently it's not an easy task!

Let's see what else other people say and maybe come up with a good solution for all!

I think the forum would be a great place! This issue of spam, unwanted posts and the like is a false issue.

Forums allow you to create private tabs, where only certain users can see and post. It is possible, with just "a few clicks", to create a tab dedicated to Bitcoin Devs, where only authorized users can post. The tab can even be visible to the general public, but only those who are authorized will be able to interact. Therefore, in this scenario, this issue of spam simply does not exist.

I believe that if devs are really interested in using the forum as a means of communication, @theymos will be happy to create a space for them quickly.

Yes.
This is what they are running: https://list.org/

There ARE some issues with it, and they are compounded by the fact that they (bitcoin developers) are running on someone else's (linux foundations) installation of it.

I really don't see it being an insurmountable task to do it themselves. But, you do wind up in the situation of is it worth it, or would it be better to use another service.

Can't really comment, since I don't contribute or mod or admin the list. I just read it....

-Dave

Is there any place where we can see their thoughts and plans on how they will move from now on? Like what options they have etc

This would be an option I suppose, although it would be more clunky for users to apply for permission before making a post, rather than the current system in the mailing list of making a post and then it being sent out after approved by the mods.

This would not work. Spam control needs to be proactive rather than reactive, or else you end up filling everyone's email inbox with hundreds of daily spam posts before they are trashed.

Yes, but this forum is pretty well known for allowing egregious amounts of spam and shitposting and never actually banning the perpetrators. A properly moderated forum could work just fine.

Well, here we are in a forum - look how much spam, shitposting, and trolling there is over here I don't think they will move it to a forum.

Github discussions, Matrix and Nostr also make moderation difficult, so they are quite an abomination for moderators.

Google Groups is clunky and is very difficult to navigate on the web.

Although one thing is certain - the mailing list is too important to let it disappear.

Why not Bitcointalk uses its forum fund to run a mailman server?

What do you mean? That they could maintain the same mailing list hosted in their own server?

Goodbye linux mailing lists! Massive contribution to open-source projects.

Actually, due to the history of this place, I think it'd be a good idea if the devs chose to host their conversations here under one of the following circumstances:

- They get their own board, in which only approved users can have write permissions. Signatures will be disabled in that place.
- They create a self-moderated thread.

This forum used to host high level discussion from Satoshi and a number of other devs when it was first created, but given the amount of unmoderated spam and shitposts on this forum now, it would be a poor choice.

I'm sure the moderators of the mailing list delete dozens of posts for every one which is actually circulated to the list.

Part of it is does anyone want to spend any time and effort and money doing it?

They are running mailman and they are not 100% happy with it. That's fine, but since it was on the Linux Foundations servers they were limited as to what they could modify.

If they setup their own server and tweaked the code (they are developers....) it might work for them with very few changes.

-Dave

Can they please move over here in this forum? I never visited that mailing list before, maybe once but I'm tired to talk to faceless experts here, if they come here we can know who they are. More importantly I with a few others can share our mind blowing and reality bending discoveries with them directly. Now which ring do we need to kiss to make that happen? Who is the BIG GUN in charge around dev team woods?😉

In fact a week ago I tried subscribing to the list but it was unresponsive. I got no email whatsoever. This explains why I guess.

Will we be able to check on the other mailing list?

I go through the link I read to have more experience, and I reach a section where the Linux Foundation has tried to move all mailings from Mailman2 to Mailman3, but they end up giving up due to scalability issues in 2017, which is something that needs to be considered because if we have issues with the scale, I don’t think we will get what we wanted.
Though this link was so educational and I also know more from that pass, I will say a great kudos to o_e_l_e_o for this great educational link.

https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2023-November/022134.html

The Linux Foundation are ending their support of mailing lists. They have agreed to continue to host the historical content, but the bitcoin dev mailing list will need to migrate to a new platform.

Options raised so far include another mailing list host, Google Groups, groups.io, Nostr, Matrix, a forum, GitHub discussions, or even do nothing and just let the mailing list end.

Either way, apparently a decision will have to be reached very soon - before the end of this year.