Topic: Six Card Rig x6 7950 MSI TWIN FROZR 3GD5/OC (Read 8179 times)

Hi guys.

I have some of this motherboards and I cannot run more than 4 280x with BAMT. I have USB Risers and any other thing seems to be ok.

Can you share your BIOS special settings?

Update:

Hephaestus was able to help me get them going in Ubuntu, discovered one of the cards was actually bad and possibly a second one.

Currently have x5 7950's stable in Ubuntu 12.04 at about 670 each temps are all below 70c one of the cards likes to spit out a handful of HW though, so I'm having that one replaced by amazon and the 6th card is also sick/dead/faulty so they both will be replaced, and I'll post back here and see where I'm at.


It would be nice to run them in windows strictly for access to afterburner, but if the lepa 1600w can handle them with under volting it, then I would much rather stick to Ubuntu or any linux based OS, SSH and byobu are glorious. 

6 Cards in Win8, I did this on a fresh install and can confirm it works.
https://bitcointalksearch.org/topic/1-btc-bounty-claimed-get-6-radeon-7970-installing-and-mining-in-windows-193695

Also for anyone else make sure to know if your board needs a presence short on any of the PCI-E lanes, my current board did need one to get the 6th card working.

https://bitcointalksearch.org/topic/bountycan-not-get-board-to-see-more-then-2-gpus-102547
https://bitcointalksearch.org/topic/list-of-motherboard-that-require-shorting-pin-a1-b17-for-pci-e-1x-to-work-36061

I've got a rig with six 7870 XTs and one with three 7950s right now (both using this same motherboard)...and I'm actually having random reboots on the one with the 7950s.    Haven't nailed that down yet.
Both rigs have 4GB of RAM and are mining scrypt just fine.

From the beginning:
Booted Ubuntu 12.04.2 LTS from a bootable flash drive, then plugged in a 16GB flash drive and installed to it like a hard drive.
Ran all the system updates after it finished installing, rebooted.
Installed aptitude (personal preference), openssh-server, byobu.
Went to System Settings -> Additional Drivers and installed the beta drivers from there, rebooted.
Can't remember if I had to run aticonfig for anything at this point.
Copied over the a prebuilt cgminer (2.11.4) for x86_64 and a shell script from another mining rig.
Apart from the random reboots (which may still be a power issue), everything works fine.

Run lspci | grep VGA and it will show you all the graphics cards recognized.
Look in /etc/X11/xorg.conf and make sure they all have entries with the correct PCI addresses (unplugging a card will change other cards' addresses).
If they don't match up, try sudo aticonfig --adapter=all --initial and then reboot.

Tried windows 8, saw 5 cards, gave error 43 for the 6th card. Took the 6th one out and couldn't run cgminer anyways. If I put it to I 10 or 11 It would maybe start for a moment, drive would crash a few times, maybe recover and then crap out, anything higher than that in I would just immediately freeze and lock up and then crap out. You would see the hash rate for everything jump 500 550 600 then freeze, everything crashes. I have a 1600w psu so I know power isn’t the issue, I was tried 13.1/28 and 12.8/2.8 in windows 8, both had the same result.

I've been in Ubuntu for the last 5 hours reading through all the documents on cgminer, despite what many believe about the system ram/gpu ram corrolation, either GPU MAX ALLOC PERCENT 100 isn't working for me or I need more memory. A high thread concurceny will crash all my gpus but one, and turn them off returning error 5 and disables them, but if I lower the thread corrurency to 1792 opposed to 21712 and turn the I down very low to 9/10/11 I can maybe get all 5 cards going at like 80-120kh/s (scrypt mining) so I'm very tempted to grab some extra dd3 memory and see if thats the issue.

But before I did that, I figured if ram is the issue than in theory I should be able to run only two cards both at full speed no problem. Pulled all the cards but two, one of them was in a pcie and would not post as enabled with hardware monitoring enabled, no matter what, removed drivers, reinstalled , rebooted, re initlized, changed cards and ports (different 1x port) same problem. moved over to a 16x port, no problem saw the card with hardware montioring enabled, Ok so I run cgminer, one card works fine, pulls 620-620, other card returns an error and doesn't hash anything. So I think the more ram theory is out the window.

Hephaestus, do you think you could make an image of your ubuntu 12,04 build that is currently running with x6 7950's? And upload it to a file storage site (ryushare, new megaupload) etc

Or provide any more details or insight to how you got that running? Out of curiosity how much RAM were you using? And you said you had it split on two PSU's ?

Thanks

Put windows 8 and you can do 6 cards.

At this moment I have 3 on powered risers, and 2 that aren't on powered risers, just risers.

I'm testing in a fresh install of win 7 64bit.

Device manager see's all 5 and they are intialized and regonized in afterburner and cgminer, however when I started up cgminer just now, the display driver crashed and then recovered. all the cards showed 0kh/s

So a little bit of progress seeing all 5 cards, now to just make them work in cgminer.

How many powered risers are you using?
I was having issues with three cards on unpowered risers, adding a powered riser (with the line to the board cut so it only pulled from the molex) solved it.

Thats interesting, I cycled over to an Xubuntu instal but it seems that even my clean installs are getting corrupted, I open firefox and terminal and the whole system just locks up and craps out. I've pulled a card and only using x5 now to eliminate the possibility of using to much power. Same issues still occuring. I Had windows 8 running on another hard drive and it saw all 5 cards fine, installed drives, booted up cgminer conncects, reports hash rate, and screen goes blank no signal all control lost, no idea.

I'll give the 12.6 drivers a go. I was using the 13.1 previously and would start cgminer and then just loose the display shortly after. They would report as mining but on the pool side it didn't show anything.

Use 12.6 drivers if you are running more than 5 GPUs per mainboard. Most other drivers will not utilize but only 4 and will disable the others. The 13.1 drivers should work but over at the AMD Developer forums people have posted the 13.1 drivers have a memory leak.

Some mainboards will need to be shorted with a thin wire on certain PCIe slots. Some boards need specific PCIe slots populated to run said amount of GPUs. There is a list of those boards somewhere on the forums. 

I was setting up three 7950s today and actually found that I need the beta drivers (from the additional drivers in the settings menu) in order to use multiple cards.  That fixed most of my problems.
After all the cards were working properly I frequently got reboots right after launching cgminer, I managed to fix that with a powered riser...
1600 watts on the output side of the power supply would be cutting it really, really close.  My three 7950s right now are pulling 940 watts from the wall (pre-tweaking).

Thanks, I use the latest AB beta, SDK 2.8. The GPUs are stock & undervolted on powered risers 4x 16x slots, 1 1x slot with 2 1x risers connected together powered closest to the GPU. 

1070/1250 @ 1.063v using --thread-concurrency 22400 -I 20 for 655Kh/s x5.

Looks good! 12.6 drivers, I'll try those. Any SDK install you did as well? and bios flashing? msi overburner? Any install details? or you just plugged everything in, clean install of windows 7 64bit, installed 12.6 drivers. booted up cgminer and off you went?

Thanks

My 5 R7950 rig runs great, GA-990FXA-UD5, NEX1500 12.6 drivers, Win7 64 bit @ 1165 watts total system.
[/URL]

Hey So I tried just installing the proprietary drivers like you did and running cgminer, and cgminer spits out GPU 0 failure, disabling, for all the cards excpet one, and so then they all list in cgminer as OFF. I followed a few different guides on getting cgminer up and running in ubuntu/xubuntu and no luck. All the cards just show as off, expect for one. And then a few seconds in my display will go blank and I'll loose picture. Computer still on and running, no display. An ideas whats going wrong? I don't think theres anyway that 1600w wouldn't be enough for all 6 but is it possible?

Suggestions? I really just want to get this damn thing stable. I've had all the parts for almost 7 days now -___-

You've got it to easy my friend lol, all 6 cards are plugged in, pci slot is presence shorted, popped in my ubuntu disk, lets see where I am in an hour.

I just installed the drivers from the settings menu under "Restricted Drivers", it will download and install them automatically from the repository.  I had to install the beta ones for my 7870 XTs though.  I'm not sure what version number any of them are...so I feel kinda dumb about it. 

I haven't actually tweaked the BIOS yet, and no dummy plugs needed.  Nothing is plugged into any of my cards, I do everything over SSH with byobu so I can detach sessions.

Right ok so then I just followed this diagram

http://blog.zorinaq.com/images/pcie-short-schematic.png

and did that to the third pcie 1x slot, thats the only thing thats need to be presence shorted

Yeah windows just craps out in so many departments though, remote desktop client wouldn't work, updates forever and ever, system instability, I'll give both a go and see what happens. Which drivers did you use in Ubuntu?

And what BIOS settings did you tweak? Also, are you using dummy plugs? actual plugs? no plugs?

Thanks

Sorry for the confusion, if you're using x16 to x16 risers, you shouldn't need to do the presence short for the x16 slot like I did.  I had to do it because I had the graphics card there on a riser like this one: http://www.amazon.com/Micro-SATA-Cables-PCI-E-Riser/dp/B005G4SADY/ref=sr_1_1?ie=UTF8&qid=1370553268&sr=8-1
So it was only connected as x1 even though it's plugged into an x16 slot.

And yes, the farthest x1 slot from the CPU is the one that needs the short.  I actually just peeled the presence lines from the risers, clipped and soldered them together because I didn't have any wire that fit very well.

Undervolting is harder in Linux, or so I've heard.  If cgminer can't do it you're pretty much screwed...
The real advantages of Linux are 1) free and 2) can be installed to a flash drive.

You have the same motherboards!!! YES!!!! With 6 running?! *hallelujah* That literally just relieved me so much, because I really didn't want to go through the hassle of having to order a different Mobo/CPU setup.

Ok, presence shorts on the thrid x1 slot assuming your counting the northbound (coming from cpu moving down the board) one as #1 then pcix16 then pcix1 # 2 and pcix1 # 3 and pcix16 pci 16 end of board

So I need to presence short pcix1 #3 and pcix16 just under it? Got it. Could you send me pictures of your presence short? I've seen pics on line for a 1x slot, but not of a 16x, or if you could link me to a good pic/diagram.

I have powered x16 to x16 extenders, I have powered x1 to x16 as well, but they were done incorrectly and don't provide the extra 12v I think... None the less i have the 16 to 16 which I know were perfectly, and since I'm using one PSU and its the same one for the mobo I don't need to sever any of the lines.

Yeah I have these MSI cards and they undervolt just fine in windows so hopefully it won't be much of an issue in Ubuntu, any advantages to Ubuntu or Xubuntu?

Any chance you mind emailing or IMing me or some kind of instant communication and we can discuss a little more?

Oh and in regards to the song my gf wrote lol, we live in these new apartments for college students and our walk in closet is actually through our bathroom, so you walk into the bathroom and then theres another door to the closet which is directly in front of the toliet and often open, and in there on the top shelf under an AC vent are all my rigs Lol so she had a clear view of them

I've got two of the exact same motherboard you have, and I'm running six cards each as well.  Using Ubuntu 12.04.2 LTS.  I can't really say anything about Windows vs Linux, I've only been mining in Linux.


Yes, you do need a presence short on the 3rd PCIe x1.  I've got all my cards on x1 to x16 risers (even on x16 slots) and had to do the presence short on that last x1 slot and the x16 slot just under it.
There aren't any additional power lines on the longer PCIe slots, x16 won't have any more than x1 does.  Take a look: http://en.wikipedia.org/wiki/PCI_Express#Pinout - all of the power pins are on the little section before the notch.
On my boards, I use three unmodified and three powered risers.  With my powered ones though, I sever the +12V lines going to the motherboard so that it only pulls power from the molex.  I actually have those powered risers powered off the second power supply.
If you don't sever the lines, you have to plug those molexes into the same power supply that's powering the motherboard.


1600 watts should be enough, I measured one of my rigs with a Kill-A-Watt at about 1500 watts with all six cards running (before playing with voltage).  I actually over-spec my power supplies, each rig has two in the 900-1000 watt range.


There's a mining rig in your bathroom?

Oh sorry forgot the mobo model. Its this one GA-F2A85X-UP4

http://www.amazon.com/Gigabyte-Motherboard-Triple-Monitor-support-Windows/dp/B009L1EOKQ/ref=sr_1_1?ie=UTF8&qid=1370547565&sr=8-1&keywords=gigabyte+fm2

Ok good to know, keep me posted when you get 5 or more running stable, I'd like to know about the details.


Yes it would be more stable, but a simple script to start cgminer when the computer reboots, and a setting in motherboard bios for it to always be on takes care of constant reboots, though I am trying to avoid that all together. And it is extremely more profitable to run x6 cards or more in one rig. The more cards in one rig, the less you spend on infrastructure (mobo,cpu,ram,shell) per card. Which improves the ROI a lot as well.

If I can get 6 stable and it takes me a whole weekend of testing to get it, it'll be worth it. But I don't see your point, it is becoming a huge headache and I've put in(wasted) a lot of time as it is.


I actually have thought about that, I would imagine the bios has an option to downclock it, which would use less volts if I couldn't force a lower constant voltage. That is if were speaking about the memory for the system and not each cards actual memory, the cards stay plenty cool. The mobo/cpu/ram also stay pretty cool. So I don't think its that, but I suppose its worth trying, have you seen info/post anywhere that recommend this? or confirm any kind of results?

 

Theoretically, GD65 should run 7 cards.
https://bitcointalksearch.org/topic/motherboard-of-choice-for-6-or-more-gpus-186877
https://bitcointalksearch.org/topic/1-btc-bounty-claimed-get-6-radeon-7970-installing-and-mining-in-windows-193695

Can you undervolt the memory as well? Maybe it is too hot.

I honestly think you'd be better off just running 4 cards in one rig, way more stable. Trying to push to the limit is fun but you soon get tired of constantly rebooting your rig etc. I like sleeping well and not worrying if it crashed 3 minutes after I went to bed 

What motherboard are you using?

I run Windows 8 w/ 13.1 & 2.8

Im running a Lepa G1600 and working on trying to get 5 - 7950s working, up to 4 ATM, on it

Need as much feedback and advice as I can get!!

I'm trying to get a x6 7950 up and running smoothly. I've been doing some preliminary testing on different, OS's, tried, BAMT, CentOS, Xubuntu, Ubuntu, Windows, I've cycled through all of these a million times. Originally I had a 850w corsair and 600w corsair, but that was a no go. Tried a 1000w corsair and 600w corsair still no go, tomorrow I give it a run with a 1600w Lepa, that'll solve my power promblems hopefully.

I'm running MSI 7950's the R7950 TWIN FROZR 3GD5/OC to be specific. Now with this card under volted at 1.087 and many other settings in cgminer, I can easily hit 680, with close to 12% reject shares in Windows. In Xubuntu it was about 615 12% stale, in Ubuntu with a higher clock/memory about 650, with again 12% stale, and so this was just testing one card, but as soon as I plugged in a second or third card Windows would lock up with driver issues. Ubuntu/Xubuntu would have power issues. (with the 1000w) with the 850w I could see two fine in any OS, and could mine with both pulling 680 with 12% reject shares.

I was using 12.7, 12.8, 13.1 drivers, tried all three, 13.1 seems to be doing good right now. And for the SDK 2.8

Now for the issue of PCIe slots, all three of my 16x slots handle non powered and powered extenders fine. my 1x slots one and two are good, but the third one I think might need a presence short, it was being picky. As far as powered extenders go, I was planning to do something like all the 16x powered(three) and 1x (three) non powered. This should be ok I think, or do pcie 1x pull less power through the board and would benefit more from a powered extender? And or should I just power extend all 6 of them?

As far as cooling goes, I can keep any card at 680 hashing constant, and keep the card below 65c with fans at under 60% (Not bad right?   ) So cooling isn't an issue. I have an open rig design, custom aluminum rig that I build, all the cards spaced out and bolted in and elevated. Excellent air flow/cooling etc. So heat from the GPU's isn't an issue.

Now do you think a 1600w 80 plus gold psu the Lepa one, that should be able to handle x6 cards just fine. What OS would you recommend for the best hashing? I really enjoy Xubuntu and Ubuntu, I just need to be sure to get up there with the hash rate and low rejected shares. I'm mining a lot of new little scrypt coins on small pools so I thought maybe it was the pool? I threw in some settings into cgminer, scan time, query, and one more I forgot, combined with a lower intensity of 18 opposed to 20, that were able to push the rejected shares  down to close to 5% however I was only mining at 650 (this is in windows)

So what are recommendations on OS, PSU, pcie slots/extenders, drivers?

Thanks,

Greg