Even more nodes deployed!...
jeebus dude, makes my 9 nodes seem like childs play. though I did set up some very high horsepower beasts (8GB RAM and tons of network bandwidth on 1Gbps ports)
how on earth do you manage them all? mine require frequent upervision to make sure they dont get stuck on a block or something
I've been working on monitoring, tools and automation. It's still primitive, but will improve after I get the 100 nodes all up.
Some simple scripting can go a long way with identical servers (even without using cssh or capistrano). For example, to restart all of them:
Code:
$ for i in {1..70}; do ssh user@node$i.nxtbase.com -t -C "sudo supervisorctl restart nxt"; done
So, if you take that pattern and combine it with some high level scripts, you can do a lot with basic primitives without burning time on higher level tools. Once the 100 are up, I'll start writing monitoring scripts for "stuck block" and other problems to have a "self healing" kind of system. My approach is to push toward scaling problems quickly and then start playing whack-a-mole on the suck.
