Server room rack power

[cpvh]

Ok, not strictly D-I-Y i know, but I have been asked to come up with some specs so that we can get quotes for a new power supply for our server room at work.

W have 3 tall racks (30U from memory) round about 6 - 7 feet anyway which are mostly stuffed full of Dell servers. These are rated at 2 Amp (The 1U jobbies) and 5A (2U jobbies). I don't know if this is the maximum they draw at start up, or if they may take more at startup. What I do know is that the 30A breaker which feeds the radial circuit they all go back to (via a few UPSs) trips out if the power comes back on after an outage

This can be a bit embaressing if we are all asleep in our beds and there is a power cut. The sequence goes something like

Power goes off,

UPSs continue to power equipment

Backup generator kicks in

MCB opens

UPSs keep the servers up for 10 or 15 mins

Then everything goes down.

If we are here when it happens, we just put the breaker back in while the UPSs are holding up and live to fight another day.

Anyway, to the question (I hope I haven't asked this before, I couldn't find it via google) Is there a rule of thumb for specing power supplies for these racks? Based on the rating plates, I could allow 2.5A per U, but I don't know if I need to allow extra over what it says on the rating plates of the servers, or if Dell have already done that. Does 60A per rack sound about right, way over the top? Way under?

Assuming it is right, what would be the best way to provide it? A number of 16A or 30A Industrial sockets?

Answers on a postcard please. :+))

[Andy Burns]

large inrush current from switched mode PSUs

I'd be tempted to split the load across multiple 16A outlets, to reduce the total inrush current per circuit, also switch to type-C MCBs.

Alternatively consider intelligent power distribution strips which can sequence the servers back on after any power cut, and have the advantage you can remotely power them off/on if required.

[Tim S]

Andy Burns coughed up some electrons that declared:

Agree. At Imperial College, we had this problem - had to go to type-D breakers in some cases, but the design considerations for putting type D in are onerous (eg you need a *very* low L-E impedance back to the supply: fortunately the 11kV transformers were round the back of the building and the College has a resident Electrical Engineer to worry about these things).

APC masterswitches were good at this, and allowed for remote power switching via RS232 and ethernet. Think that range has been superceded, but it's likely APC still make something similar.

Cheers

Tim

[Pete Cross]

I'm assuming the ups's keep the pc's running and the breaker supplies the ups's ?

Therefore isn't it more likely the ups's overloading the circuit when the power is returned ? they can have quite hefty startup currents, especially if the batteries are already under load and discharging rapidly.

Pete

[Andy Burns]

Usually in a rack setup like this each server has two PSUs, one supplied direct from mains (with generator backup) the other supplied from mains via UPS.

There is no initial disruption to the half of the PSUs being fed from the UPS, so no surge, when the mains goes off there will be a surge when half the PSUs come back on line when either the generator kicks in, or the mains is restored.

If this surge trips the MCBs then you leave the servers on one PSU permanently, if this UPS happens to be on the same MCB they'll all die when the battery is exhausted.

[Pete Cross]

I see now then...

So what you need is a delayed start on the psu's, same as on a sunbed where the tubes are powered sequentially otherwise it would exceed the 13A fuse in the plug.

Pete

[Dave Liquorice]

Both, I'm surprised racks with so many servers don't already have master switches just for the sequential start up, the abilty to power cycle individual bits of kit without having to physically visit is just a useful side effect. B-)

As for power, per rack 60A is 13.8kW, 3 racks 41.4kW I hope you have some good air con! Maybe buying a decent clamp meter and measuring what is actually being drawn would be useful. Not sure how they are affected by distorted waveforms produced by loads a SMPSUs though...

[Peter Crosland]

It sounds like a classic case of your employer wanting to do it on the cheap the result of which could be that you get the blame when it all goes pear shaped.

The whole job needs to be specified by a specialist professional in the field, which is not likely to be you average sparky who, with all due respect to inhabitants of this NG, will no experience in this sort of installation. If possible get three different quotes for the specification and the installation of it. Get references for similar work they have done and take them up.

Peter Crosland.

[Andy Burns]

You can get "dumb" sequential startup strips too

Which you may not need if your servers have "lightsout" control where you can turn them on and off over a dedicated network which is powered from the standby power of the PSU.

Also with servers on multiple power supplies you need to have multiple master switches and power off all relevant outlets feeding a single server to reboot it.

[dmc]

We have 50 odd racks in our machine room - and often had this problem.

The Dell servers were by far the worst culprits along with the EVA storage. Sun and HP servers tend to be better. We have dual psu machine feed from two independant distribution networks and most Dell boxes appeared to run from one or other psu on not share the load between them. This tended to cause one side of the rack to be overloaded and cause the breakers to pop :-(

Sorted by fitting different MCBs in the SAN racks and decent sequencing power distribution to the server racks. We use APC power dist (I think the models we have are

) which give us remote (ssh or https) control of the outlets and also sequencing (can be configured with timings to allow things to boot in order). They also offer some load monitoring.

The ones we have are zero U models that fit nicely into our APC racks - you might be better with a few of the rack mount versions if you don't have room to mount them.

Not cheap, but pretty solid

Darren (oh, we stopped buying Dell servers as well :))

[dmc]

Or even better, both feeds from UPS protected feeds, with gen backup.

Of course, then someone buys a bloody sun server with one power lead or even worse, one with *3* power feeds.

Stupid bloody machines. Thankfully, I think sun have stopped that now :)

Darren

[mick]

BAe Systems at Warton had a lot of problems with circuit breakers feeding computer suites. They eventually solved it by going back to fuses! We supplied them with some rather nice (if I may say it myself) fused distribution panels.

[James Salisbury]

Try replaceing the power distribuiton units with

so not everything turns back on at the same time.

I do also hope you have high integrity earthing as well.

[Andrew Gabriel]

Well, the whole point of splitting the power is so that if one supply fails, the other continues to run the servers. If either supply isn't capable of doing that, then there's no point in splitting the servers across multiple supplies.

[dmc]

Yes, but the start up surge on the dual psu dells was all on one psu. Given we had plenty of them it meant seriously under speccing the power feeds and hoping they wouldn't all come up and suck power from the same feed. It was fairly random IIRC (at least if it was consistant which psu they would pick we could have shared them about).

Sods law meant that 80% of them would come up on one of the feed tripping it out are which point they would all switch to the other feed and the surge would trip that.

As I say, we switched to HP for winblows servers and it got a lot better (maybe just better psus?) and now we have all the racks sequenced anyway so the problem has gone for good (he says, touching wood/mdf :))

Darren

[cpvh]

Good stuff as he usually does.

Many thanks one and all, and apologies for not being around for a couple of days to read the replies. It has given me lots of food for thought.