Server Closet in a Condo - How would you build it?

[svtkobra7]

Hi,

I was referred to this forum by a good friend to ask a specific question; however, I'm really glad he made me aware of it as I've already learned a lot perusing the content here!

I wish I had been aware of this forum during a project to turn a coat closet into a server closet. Note the vertical mounting orientation was required because the servers are too deep for the closet (horizontally). Also, I should note that I beefed up flimsy metal studs primarily because directly behind the closet exists a custom closet I installed. While plywood on a french cleat would probably have been OK without the additional 2x4s and 2x6s, I figured it better to over-engineer initially instead of figure out I have a problem later.

The point of my post - while my solution works - I'm nowhere near as knowledgeable as forum members here - and I'm just curious how you guys would have done things differently / where I screwed up? (Again, I'm quite fascinated with the content here!)

Here is what I did:

• Removed drywall on all sides of closet.
• Bent lips of steel studs to allow insertion of wood 2 x 4s (verticals) and fastened them in place.
• Added wood 2 x 6s (horizontals).
• Inserted Rockwool SAFE'n'SOUND sound abatement insulation on all 5 sides of closet. (Servers belong in datacenters and get noisy)
• Ran armored electrical cable to the ceiling and side, and installed a bathroom exhaust fan on the ceiling, and switch of course. (Servers exhaust serious heat and it would overheat or run too hot to my liking without proper airflow).
• Replaced drywall.
• Used a french cleat to add a plywood panel.
• Mounted vertical server racks to plywood panel.
• Increased gap under closet door to 3/8" to allow for air inflow (negative pressure environment created thanks to the exhaust fan, even at a paltry rating of 110 CFM).

Part of my thinking here was that when I sell my condo, minimal effort is required to revert to salable condition. The plywood panel can be removed simply by lifting it vertically, the french cleat on the drywall is easily removed and only a few holes need to be patched, and I doubt most people would even catch the fact that the "light" is an exhaust fan (and probably no need to remove).

 

[greenlizard]

Whoa! Nice work.

 

[svtkobra7]

Whoa! Nice work.

Thanks - It definitely was a bit of work, for sure. Curious how others would have approached!

One item I can attest to (in hindsight) is that enterprise grade servers belong in data centers, not in homes, and definitely not in condos where you are space constrained. But if you have to have servers in your home, proper attention must be paid to air flow and sound abatement, otherwise your home just became a datacenter due to the heat and noise.

About the only thing I would do differently would be to install a solid door, to absorb more sound than the current hollow one, but I'm not sure how beneficial that would be as I rely on the gap beneath the door for air inflow, so I can't ever completely seal that room.

One bit I find quite interesting, despite such a sub-optimal installation, and the fact that the closet doesn't get the best airflow (under the door only) ... simply due to creating a system with air flow (i.e. the closet, with defined intake/exhaust) and aligning that with how servers intake and exhaust air, I achieve lower HDD temps in that closet than if a server is sitting in my office horizontally (temporary as I work to solve a fastener issue).

BTW - As a rule of thumb, HDD temps shouldn't exceed 40C (although there are many differing opinions on that).

Server in Office max HDD temp over last 24 hours = 38C
Server in Closet max HDD temp over last 24 hours = 32C
(see attached)

[/end being a ****]

 

[Provincial]

My only experience with controlling heat in electronics is maintaining aircraft, which have some radio gear. This includes installing a complete avionics package in my Cessna, including a system for cooling. My comments are general, and I don't have the knowledge to get into specifics.

First, using a bathroom fan is a great idea to prove the concept. Cheap, relatively easy to install, and fairly quiet. Most bathroom fans are designed for intermittent use only, and you can expect a limited life if run full time. They commonly do not have ball-bearing motors. They also move a lot of air, which means you will be replacing that exhaust air with conditioned (heated or cooled) air from the condo. It costs money to condition that air, unless your outdoor temperatures are in the low 70's year round.

Consider building a shroud around the server. This could be just a sheet metal channel that is open on the bottom and has a duct attached to the top end. Design the mounting to allow easy access to the server using hinges or keyhole mounting holes. The idea is to direct the moving air close to the server, since air that does not pass close enough to the server to carry away heat is wasted and has to be replaced by conditioned air.

If you use a shroud, you will not have to move as much air. This would allow you to use a smaller fan, perhaps a large "muffin" fan, which will be quieter than the bath fan. These have been used to provide air flow for cooling Variable Speed Drives, and even to provide extra cooling for 3-phase motors running at very slow speeds using VFD's. It may be possible to mount it to the shroud and run a duct to the current bath fan discharge point. If not, mount it where the bath fan is now and run a duct from the shroud to the fan intake.

You will need to move much less air if you use a shroud. It is beyond my skills to determine a design parameter, but a desktop gaming computer doesn't take a very large fan to provide proper cooling. Fan size will determine duct size.

The shroud will also be another chance for noise abatement. Attach sound deadener to the outside of the shroud. The intake could perhaps be baffled with sound-deadening material to make it even quieter.

If the discharge air from the fan isn't noisy, you could duct it back into the condo during the heating season, recovering the heat energy of the server and putting the energy of the fan to use in circulating the warm air.

I am very impressed with your project. You made a very clean installation. I am just trying to tickle your imagination on how to make it quieter and more energy efficient.

 

[Ole Slewfoot]

A baffle made of carpet creating a Z shape path without line of sight from the rack to the door gap will help with noise, as would carpet egg crate shaped wall hangings.

Since servers don't smell bad, I might look at a way to exhaust into the home during winter

 

[bradpac]

Your installation is great, especially with the goal of ease of turning it back to a normal closet. For me, I probably would have made everything free standing on legs just because drywall and wood work is not something I enjoy. I would have rather welded up a rack on some legs and shoved in it there, but it would not have turned out as clean of an install as what you have.

The bathroom vent is currently how I exhaust our communications closet in our little office, the gap at the bottom of the door is an important detail. Good job! I keep our servers in a wall mounted, enclosed rack out in the warehouse that has its own dedicated a/c duct with thermostatic exhaust fans, works well. They didn't want the noise in the office, and to a metal roofing manufacturer the computer equipment placement isn't a priority, I fight for what room I get.

 

[svtkobra7]

Your installation is great, especially with the goal of ease of turning it back to a normal closet. For me, I probably would have made everything free standing on legs just because drywall and wood work is not something I enjoy. I would have rather welded up a rack on some legs and shoved in it there, but it would not have turned out as clean of an install as what you have.

• I genuinely appreciate the compliment and the acknowledgement of ease of reversion back to a standard closet. I suppose when you start with a blank slate, or bare studs better stated, you can "engineer" for such.
• I really wish I knew how to weld / had the equipment!

The bathroom vent is currently how I exhaust our communications closet in our little office, the gap at the bottom of the door is an important detail. Good job!

• Agreed on the importance of the gap between the floor and the bottom of the door.
• It would be interesting to "plug" that gap and see what happens to HDD temps, monitoring remotely, in fact I will give that a try just out of sheer curiosity.
• Considering I added weather-strip to the other 3 edges of the door, to aid in sound abatement (and indeed it did have a discernible effect to my un-calibrated ears), the servers would be "choking" for air.

I keep our servers in a wall mounted, enclosed rack out in the warehouse that has its own dedicated a/c duct with thermostatic exhaust fans, works well. They didn't want the noise in the office, and to a metal roofing manufacturer the computer equipment placement isn't a priority, I fight for what room I get.

• The full-depth enclosed racks I've seen (or maybe just that I recall), all orient servers vertically, but in a right-side up orientation, i.e. chassis front facing ceiling (opposite of my "upside down" approach) … is this the case with your installation?
• I ask as when I migrated from a single server to two, I oriented one with the chassis facing the ceiling, and the other with the chassis facing the floor, to compare the impact on HDD temperatures.
• [Let me interject a few particulars such that my following comments have more meaning. My particular motherboards allow the user to select 1 of 4 "fan modes" (Optimal, Standard, Heavy I/O, and Full, listed in order of increasing RPM / noise) and I wrote a script, which executes every 60 seconds, and performs the following tasks: (a) polls all 12 HDDs for their temperature, (b) selects the quietest fan mode needed to ensure the warmest HDD is below 40C, (c) if needed, issues a command via IPMI to change the fan mode, (d) logs min/avg/max HDD temperature, whether fan mode was changed, and if so to what fan mode.]
• As to the results if you are curious, over a 24 hour period:
  • Server with chassis front facing ceiling: Average HDD temp = +5.3C warmer than other server, fan mode used nearly the entire time = Full (loudest / 100% duty cycle). In fact this server couldn't hold HDD temps to 40C or below even using the fans at full speed (assumption is that this was caused by intake of "recycled" air being exhausted by both servers - air intake @ chassis front / exhaust @ chassis rear of course).
  • Server with chassis front facing floor: Average HDD temp = -5.3C cooler than other server, fan mode used nearly the entire time = Optimal (quietest / lowest RPM).

 

[svtkobra7]

A baffle made of carpet creating a Z shape path without line of sight from the rack to the door gap will help with noise, as would carpet egg crate shaped wall hangings.

• Thanks for offering ideas for sound abatement (my fiance was mentioning tonight she always hears a "humm" - I don't hear it).
• Could you expand a bit on the baffle idea please, I can't picture it in my head?

Since servers don't smell bad, I might look at a way to exhaust into the home during winter

• That bathroom fan doesn't actually exhaust externally. Rather to a ~600+ cubic foot "enclosure" below the concrete slab ceiling, above the closet ceiling, which is segregated from the rest of the "air space." (authoring a reply now in response to @Provincial that will explain more accurately).

 

[Jawn]

My approach would probably have been a small freestanding rack surrounded with sufficient sound deadening and perhaps some forced air ventilation and/or cooling as needed.

But, nicely done. Nothing wrong with what you've done, far as I see.

I'm curious what the use case is for the servers (if you don't mind sharing)?

 

[rlitman]

My approach would probably have been a small freestanding rack surrounded with sufficient sound deadening and perhaps some forced air ventilation and/or cooling as needed...

APC sells one of those. It's quite good at taming the noise, but is also pretty expensive.

• 
  ...
• The full-depth enclosed racks I've seen (or maybe just that I recall), all orient servers vertically, but in a right-side up orientation, i.e. chassis front facing ceiling (opposite of my "upside down" approach) ...

• 
  
  Your "upside down" approach is absolutely correct for thermal management, but is difficult, because most systems are not designed to be hung in that orientation, and gravity being what it is, makes computers stood up vertically a little unstable. Still, I like what you've done.
  
  As for the door, a solid door helps a little, but not as much as you might think, as wood conducts sound pretty well. A fire rated door (gypsym filled), or a commercial solid slab door (chip board) might be a little better. One option would be to hang a sheet of MLV behind the door.
  
  The gap under the door should be at least equal in area to the duct size exiting the fan. Your 3/8" gap seems to be an ok minimum to match your fan's CFM, but I think that your fan is undersized for the job. You could also cut the sound some more by adding something to the back of the door and the floor to bend the airflow around a serpentine path.
  
  But the biggest cause of noise will be because of the heat. Modern computers speed up their fans (greatly increasing the noise) when their temperature gets too hot. Keep them cooler, and they run much quieter.

 

[svtkobra7]

I am very impressed with your project. You made a very clean installation. I am just trying to tickle your imagination on how to make it quieter and more energy efficient.

• Thank you very much for the compliment and your very thorough response (sincerely appreciated).
• Your response is exactly what I had hoped to elicit by posting the installation, and I love the proposed optimizations here!
• Generally speaking, I think there may be some marginal enhancement to sound pressure level (although quite "tolerable" all things considered), but as to the energy efficiency, while I'd love to pursue (and I'm inclined to), even if I managed to reduce the power consumption of fans managing air flow in the closet (both the ceiling fan and fans in the chassis) to 10% of current levels, that saving is still a fraction of power consumption of the servers (as i write this consumption = 738 watts).

First, using a bathroom fan is a great idea to prove the concept. Cheap, relatively easy to install, and fairly quiet. Most bathroom fans are designed for intermittent use only, and you can expect a limited life if run full time. They commonly do not have ball-bearing motors. They also move a lot of air, which means you will be replacing that exhaust air with conditioned (heated or cooled) air from the condo. It costs money to condition that air, unless your outdoor temperatures are in the low 70's year round.

• Solid point regarding the life expectancy of the bathroom fan motor and indeed I went that route initially as a "proof of concept" thinking that if it burnt out quickly, I would replace with a stronger motor rater for continuous use. (cross that road when I come to it mentality)
• Surprisingly, it has been in use nearly 24/7 since summer 2017

Consider building a shroud around the server. This could be just a sheet metal channel that is open on the bottom and has a duct attached to the top end. Design the mounting to allow easy access to the server using hinges or keyhole mounting holes. The idea is to direct the moving air close to the server, since air that does not pass close enough to the server to carry away heat is wasted and has to be replaced by conditioned air.

If you use a shroud, you will not have to move as much air. This would allow you to use a smaller fan, perhaps a large "muffin" fan, which will be quieter than the bath fan. These have been used to provide air flow for cooling Variable Speed Drives, and even to provide extra cooling for 3-phase motors running at very slow speeds using VFD's. It may be possible to mount it to the shroud and run a duct to the current bath fan discharge point. If not, mount it where the bath fan is now and run a duct from the shroud to the fan intake.

You will need to move much less air if you use a shroud. It is beyond my skills to determine a design parameter, but a desktop gaming computer doesn't take a very large fan to provide proper cooling. Fan size will determine duct size.

• Restating for my own understanding … (1) I'd describe the configuration today as a system (A) within a system (B); where A = servers, B = closet, and it works since A's air flow aligns precisely with B's. (2) Taking that a step further, you are suggesting an integration of A & B, such that they function as 1 system (and not a system within a system). [maybe ... hopefully ... make sense???]
• I follow your points; however, I believe the bottleneck is the 110 CFM (even though that is 2 x the volume of the closet) fan, so I'm wrestling to wrap my head around how potentially reducing the amount of air flow moved works ... is it just that the air flow is more "directed" and "efficient"?
• (not contesting any of your remarks, just seeking to understand as I'm sure you are onto something here).

The shroud will also be another chance for noise abatement. Attach sound deadener to the outside of the shroud. The intake could perhaps be baffled with sound-deadening material to make it even quieter.

• I considered using mass loaded vinyl previously, either wrapped around the server chassis (not too different from your proposal), or simply affixed to the inside of the closet door, I wonder how effective that may be?

If the discharge air from the fan isn't noisy, you could duct it back into the condo during the heating season, recovering the heat energy of the server and putting the energy of the fan to use in circulating the warm air.

• Hopefully my description here is easy enough to follow (as I've struggled to explain it before) ...
• Concrete slabs are the ultimate horizontal "borders" above and below the condo, and in areas such as the bedrooms and living room, that concrete is fully exposed ("soft loft" architecture I suppose), yet in other areas such as the bathrooms, kitchens, and above the closet, space below the ceiling slab is "enclosed".
• That enclosed space extends 20.5" from the top slab to sheetrock and minimally is ~600 cubic feet (closet = 59 cubic feet for reference).
  
• I toyed with the notion of "tapping" into an existing vent (either from existing bathroom fan or dryer); however, didn't do so as I didn't think it was needed.
• See below image ... the red dotted line represents the closet ceiling and the horizontal arrow represents the short run of duct directing warm away from the closet, and that is occurring in that "enclosed airspace".
  
• That side of the condo is quite comfy during the winter from all of the heat from the servers.

 

[svtkobra7]

My approach would probably have been a small freestanding rack surrounded with sufficient sound deadening and perhaps some forced air ventilation and/or cooling as needed.

• I actually ran a whip down to floor level, just in case the passive intake (air under the door) wasn't sufficient, but ultimately determined an intake fan wasn't needed, and capped it put a blank plate over the junction box.

But, nicely done. Nothing wrong with what you've done, far as I see.

• Thanks.

I'm curious what the use case is for the servers (if you don't mind sharing)?

• Happy to share, down to specs if desired. To be sure, this is a hobby of mine, so I have taken it much further than needed, but the reason I migrated to my first enterprise chassis was to "rationalize" the 3 COTS NAS (think Synology or QNAP) appliances I was using, knowing that those 3 boxes + a router could be replaced with a single server chassis. So I ended up with an all in one ESXi server, running pfSense as a firewall, FreeNAS for storage, and a number of VMs for personal and work related workloads. And Plex for entertainment.
• After I blew up the ZFS storage pool a couple of times, I decided I needed another local copy of data, so I deprecated the prior server and replaced it with two.
• So that original "all-in-one server" is transforming into a fault tolerant, high availability cluster LOL (not quite there yet on the SW config side).

 

[svtkobra7]

Your "upside down" approach is absolutely correct for thermal management, but is difficult, because most systems are not designed to be hung in that orientation, and gravity being what it is, makes computers stood up vertically a little unstable. Still, I like what you've done.

• Completely understand your point, but between gravity tugging at ~125-150 lbs, that plywood is quite firmly planted in the french cleat.
• I think the fact that I used two vertical wall mount racks, to emulate the four posts of a traditional rack, and attached the servers to each rack with rails, certainly helped with the stability.
• But should I need to service the server closest to the wall, it requires removal of the one up front and additionally it is a PITA to "rack" those guys without another pair of hands.

As for the door, a solid door helps a little, but not as much as you might think, as wood conducts sound pretty well. A fire rated door (gypsym filled), or a commercial solid slab door (chip board) might be a little better. One option would be to hang a sheet of MLV behind the door.

• I'm liking the MLV idea (thanks for the info regarding sound absorption).

The gap under the door should be at least equal in area to the duct size exiting the fan. Your 3/8" gap seems to be an ok minimum to match your fan's CFM, but I think that your fan is undersized for the job. You could also cut the sound some more by adding something to the back of the door and the floor to bend the airflow around a serpentine path.

• I don't disagree that the fan could be stronger. Throwing some numbers at this ... 3/8" x 30" = 11.25 square inches intake vs. ~12.5 square inches exhaust (4" duct).
• I've struggled thinking about an intake baffle of some sort, as going that route would definitely lower the sound pressure level, but I would think significantly increase static pressure, and put that fan well out of its league.
• Each server has 3 main fans pulling air around the HDDs, with each rated at 72.5 CFM @ 1.09 in. H20 static pressure. The ceiling fan is rated at 100 CFM @ 0.10 in. H20 static pressure and 88 CFM @ 0.25 in. H20 static pressure.

But the biggest cause of noise will be because of the heat. Modern computers speed up their fans (greatly increasing the noise) when their temperature gets too hot. Keep them cooler, and they run much quieter.

• Correct ... I've taken that into my own hands by way of a script which executes once every 60 seconds and adjusts the fan mode to 1 of 4 available (using the most quiet fan mode possible to keep the warmest HDD at 40C or below). The fans are PWM, but unfortunately the duty cycle can't be controlled more precisely with this era motherboard (the succeeding generation can be controlled to increments of 1/64).

 

[Jawn]

• Happy to share, down to specs if desired. To be sure, this is a hobby of mine, so I have taken it much further than needed, but the reason I migrated to my first enterprise chassis was to "rationalize" the 3 COTS NAS (think Synology or QNAP) appliances I was using, knowing that those 3 boxes + a router could be replaced with a single server chassis. So I ended up with an all in one ESXi server, running pfSense as a firewall, FreeNAS for storage, and a number of VMs for personal and work related workloads. And Plex for entertainment.
• After I blew up the ZFS storage pool a couple of times, I decided I needed another local copy of data, so I deprecated the prior server and replaced it with two.
• So that original "all-in-one server" is transforming into a fault tolerant, high availability cluster LOL (not quite there yet on the SW config side).

Thanks!

I've got an old version of something by the same name FreeNAS (however, it's from prior to the fork that created the current "FreeNAS" product and the "NAS4FREE" which I understand has a closer familial lineage to the "FreeNAS" I have). It's on an old Artigo A2000 low power chassis. But, that's been chugging along for nearly a decade (with a drive R&R a couple years ago, so they're not that old).

Haven't thought about rolling routing/firewall into the same box as my storage, my home network is using a Mikrotik layer 3 switch that has all the functionality I need there. But I have thought about some basic *nix-driven hardware to play around with... Maybe just a RasPi.

While I like messing with the gear and it's good to stay up on what I have, I'm no expert either, so as long as it works I'm a happy camper.

 

[svtkobra7]

Thanks!

I've got an old version of something by the same name FreeNAS (however, it's from prior to the fork that created the current "FreeNAS" product and the "NAS4FREE" which I understand has a closer familial lineage to the "FreeNAS" I have). It's on an old Artigo A2000 low power chassis. But, that's been chugging along for nearly a decade (with a drive R&R a couple years ago, so they're not that old).

• A decade is impressive!
• Don't worry, you didn't miss anything with FreeNAS Corral (joke - bad sense of humor).

Haven't thought about rolling routing/firewall into the same box as my storage, my home network is using a Mikrotik layer 3 switch that has all the functionality I need there. But I have thought about some basic *nix-driven hardware to play around with... Maybe just a RasPi.

• I almost went the Mikrotik route, but ended up going with the Brocade ICX-6450. 4 x 10G links + 1 x 40G direct between hosts. This is very recent, still working on nailing everything down.
• Anyway, I copied my set up below (in hopes it sparks some ideas) ... its a shame, apparently the SPOILER bb code doesn't work ... oh well, copy / paste from another forum.

While I like messing with the gear and it's good to stay up on what I have, I'm no expert either, so as long as it works I'm a happy camper.

• Trust I'm far from an expert too. No IT background at all, but somehow with that first server I mentioned, I managed to get ESXi + FreeNAS + pfSense running in a few weeks. No prior experience to any of that software.
• I find it fun to always take things a step further and tinker, but of course, that is a good recipe for blowing something up.

Disclaimer, while not quite as bad as an all flash array that I would love, the following is massive overkill ...

Spoiler: ESXi-01

ESXi 6.7.0 Update 1 (Build 10764712) | FreeNAS-11.1-U6 | pfSense 2.4.4
System: SMCI SuperStorage 6027R-E1R12T | Chassis: SuperChassis 826BE16-R920LPB
Motherboard: X9DRH-7TF | CPUs: 2 x INTL Xeon E5-2690 v2 | RAM: 256GB Hynix DDR3 ECC 1333 MHz (200GB allocated to FreeNAS)
Backplane: BPN-SAS2-826EL1 | HBA: Integrated 2208 flashed to 9207-8i
Fans: CPU - 2 x SNK-P0048AP4 | Backplane - 3 x FAN-0126L4 | PSUs: 2 x PWS-920P-SQ
NICs: 2 x Integrated INTL X540-AT2 [@ 1G] | HP-649281-B21 (Mellanox MCX354A-FCBT ConnectX-3) [@10G / @ 40G]
NVMe: 2 x INTL Optane SSD 900P - 280GB, HHHL

• ESXi-01_Opt1: ESXi boot + Local Datastore
• ESXi-01_Opt2: FreeNAS boot + 2 x 16 GB vDisks ("Opt1log1" & "Opt1log2" as mirrored SLOG)

Bulk Storage: 12 x WDC WD100EMAZ - 10TB

Spoiler: ESXi-02

ESXi 6.7.0 Update 1 (Build 10764712) | FreeNAS-11.1-U6
System: SMCI SuperStorage 6027R-E1R12T | Chassis: SuperChassis 826BE16-R920LPB
Motherboard: X9DRH-7TF | CPUs: 2 x INTL Xeon E5-2680 v2 | RAM: 256GB Hynix DDR3 ECC 1333 MHz (200GB allocated to FreeNAS)
Backplane: BPN-SAS2-826EL1 | HBA: Integrated 2208 flashed to 9207-8i
Fans: CPU - 2 x SNK-P0048AP4 | Backplane 3 x - FAN-0126L4 | PSUs: 2 x PWS-920P-SQ
NICs: 2 x Integrated INTL X540-AT2 [@ 1G] | HP-649281-B21 (Mellanox MCX354A-FCBT ConnectX-3) [@10G / @ 40G]
NVMe: 2 x INTL Optane SSD 900P - 280GB, HHHL

• ESXi-02_Opt1: ESXi boot + Local Datastore
• ESXi-02_Opt2: FreeNAS boot + 2 x 16 GB vDisks ("Opt1log1" & "Opt1log2" as mirrored SLOG)

Bulk Storage: 12 x HGST Deskstar NAS HDD - 6TB [raidz2 6x2]

Spoiler: Networking

Switch: Brocade ICX 6450-24 (24 x 1G + 4 x SFP+)
ESXi-01

• Integrated INTL X540-AT2
  • GLAN1 => ICX 6450-24 @ 1G | ESXi "WAN" Virtual Switch + Port Group | pfSense "WAN" Interface
  • GLAN2 => ICX 6450-24 @ 1G | ESXi "LAN" Virtual Switch + Port Group | pfSense "LAN" Interface
• HP-649281-B21 (Mellanox MCX354A-FCBT ConnectX-3 fw CX354A-FCB_A2-A5 2.42.5000)
  • Port 1 => QSFP-to-SFP-Adapter => 10G SFP+ Passive DAC => ICX 6450-24 @ 10G
  • Port 2 => 40G QSFP+ Passive DAC => ESXi-02 @ 40G

ESXi-02

• Integrated INTL X540-AT2
  • GLAN1 => ICX 6450-24 @ 1G | ESXi "WAN" Virtual Switch + Port Group | pfSense "WAN" Interface
  • GLAN2 => ICX 6450-24 @ 1G | ESXi "LAN" Virtual Switch + Port Group | pfSense "LAN" Interface
• HP-649281-B21 (Mellanox MCX354A-FCBT ConnectX-3 fw CX354A-FCB_A2-A5 2.42.5000)
  • Port 1 => QSFP-to-SFP-Adapter => 10G SFP+ Passive DAC => ICX 6450-24 @ 10G
  • Port 2 => 40G QSFP+ Passive DAC => ESXi-01 @ 40G

 

[Provincial]

Thanks for the compliments.

My point on the shroud (perhaps I should have called it an enclosure, but it really isn't) is that you would be reducing the amount of air you have to move by moving only air that has been heated by the server unit. Most of the air in the closet is not being heated by the server, so if you can focus on only that air that is actually being heated (moved by the fans in the server), you can reduce the amount of air you move. Right now, the server uses it's built-in fans to draw off heat. The only air you need to move is that air. If the cooling air discharge area of the server is located in one area, in theory you can set up a shrould and duct work to draw off only that air. If the cooling air discharge is more omnidirectional, a shroud would have to be larger to capture it all.

All modern automobiles have shrouds and ductwork that allow fans to move air through a radiator so small that it would not properly cool the engine if the radiator was only exposed to a much larger inlet and had no shrouds around the fan(s). Those shrouds and ducts mean a smaller amount of moving air can provide adequate cooling.

It is inefficent, but adequate, to let the heated air rise to the top of the closet and be removed by the bath fan. That setup also moves a large amount of air that has not been heated.

Since you are discharging into your loft space, you are recovering that heat in the heating season, but in the cooling season you are having to use your air conditioning to reduce the temperature of your living space air.

 

[svtkobra7]

My point on the shroud (perhaps I should have called it an enclosure, but it really isn't) is that you would be reducing the amount of air you have to move by moving only air that has been heated by the server unit. Most of the air in the closet is not being heated by the server, so if you can focus on only that air that is actually being heated (moved by the fans in the server), you can reduce the amount of air you move. Right now, the server uses it's built-in fans to draw off heat. The only air you need to move is that air. If the cooling air discharge area of the server is located in one area, in theory you can set up a shrould and duct work to draw off only that air. If the cooling air discharge is more omnidirectional, a shroud would have to be larger to capture it all.

• Thanks for revisiting the topic .. completely understood now.
• It is indeed 1 area ... racks are 19" wide and each 1RU = 1.75", so 2 x 2U servers = an area of ~19" x 7" to capture exhaust by way of shroud / enclosure.
• I can see this working quite nicely in my head now ... I wonder how the intake can be revised with a working plan for the exhaust in play ... I like the idea of a baffle to further deaden sound, but could see that being a busted play without a straight path of air flow to the HDDs (~14" off floor, same ~19" x 7" surface area).

All modern automobiles have shrouds and ductwork that allow fans to move air through a radiator so small that it would not properly cool the engine if the radiator was only exposed to a much larger inlet and had no shrouds around the fan(s). Those shrouds and ducts mean a smaller amount of moving air can provide adequate cooling.

• Solid analogy ... I assume some of this efficiency is "free" in that the velocity of the vehicle will serve to increase the static pressure / velocity of the air moving through the exchanger as well?
• My only freebie is convection working on my side (likely to a minimal extent?)

It is inefficent, but adequate, to let the heated air rise to the top of the closet and be removed by the bath fan. That setup also moves a large amount of air that has not been heated.

• Got it.

Since you are discharging into your loft space, you are recovering that heat in the heating season, but in the cooling season you are having to use your air conditioning to reduce the temperature of your living space air.

• Absolutely, there are two sides to that coin. I keep ambient at 78F - 80F during the summer (any colder and I'm cold, frankly), it would be interesting to know the impact in F that ambient is raised by that exhaust.

THANKS AGAIN!!!

 

[alberto]

Great job.

A minor suggestion is to use QuietRock instead of regular drywall. You could go further and use some of their other products (quiet seal, quiet putty) to further isolate noise. Not sure that the hum your girlfriend hears would have been avoided with additional insulation as that noise is created by the need to ventilate (fan, gap at bottom of door, etc.), but it might have helped. You could also use a Panasonic WhisperCeiling fan, for example. They are very quiet, and although as someone noted, not designed to run all of the time, they do have a 6 year warranty and the motors use ball bearings.

 

[rlitman]

• 
  ...
• I think the fact that I used two vertical wall mount racks, to emulate the four posts of a traditional rack, and attached the servers to each rack with rails, certainly helped with the stability.
  ...
• I don't disagree that the fan could be stronger. Throwing some numbers at this ... 3/8" x 30" = 11.25 square inches intake vs. ~12.5 square inches exhaust (4" duct).
  ...

• 
  
  Agreed, and I like your solution of turning a 4-post upright. You made the most of a system that really isn't made to work that way.
  
  Yeah, my numbers roughly match yours. Though I was expecting that fan to have a 3" duct. If the door is currently just over a floor, you could easily open up the gap to 1", and install a wood threshold at a later time.
  
  

  ...
  My point on the shroud (perhaps I should have called it an enclosure, but it really isn't) is that you would be reducing the amount of air you have to move by moving only air that has been heated by the server unit. Most of the air in the closet is not being heated by the server, so if you can focus on only that air that is actually being heated (moved by the fans in the server), you can reduce the amount of air you move. Right now, the server uses it's built-in fans to draw off heat. The only air you need to move is that air. If the cooling air discharge area of the server is located in one area, in theory you can set up a shrould and duct work to draw off only that air. If the cooling air discharge is more omnidirectional, a shroud would have to be larger to capture it all.
  
  All modern automobiles have shrouds and ductwork that allow fans to move air through a radiator so small that it would not properly cool the engine if the radiator was only exposed to a much larger inlet and had no shrouds around the fan(s). Those shrouds and ducts mean a smaller amount of moving air can provide adequate cooling.
  
  It is inefficent, but adequate, to let the heated air rise to the top of the closet and be removed by the bath fan. That setup also moves a large amount of air that has not been heated.
  
  Since you are discharging into your loft space, you are recovering that heat in the heating season, but in the cooling season you are having to use your air conditioning to reduce the temperature of your living space air.

  
  That's an interesting point. By mounting the systems vertically, he's counting on the heat to rise naturally, and most of it will, but some may end up recirculating into the computer. If you can duct the exhaust fan to capture the heat straight from the computers, the room will be more efficiently cooled, and the computers will have a lower inlet temperature, which can in turn reduce noise.
  
  You could do this by putting a rectangular to round duct transition over the computers, then then run a piece of flex duct up to the fan. But how much of a difference it would make is anybody's guess. I'm thinking that with the vertical arrangement as it is, it probably wouldn't do much.
  
  Another option would be to water cool the computers. That's more effort, but could potentially cut the noise to inaudible levels.

 

[bradpac]

• 
  
  • The full-depth enclosed racks I've seen (or maybe just that I recall), all orient servers vertically, but in a right-side up orientation, i.e. chassis front facing ceiling (opposite of my "upside down" approach) … is this the case with your installation?
    

• 
  
  
  The server in our rack is actually sitting horizontal, about 4" was added onto the back of the enclosure so it would fit, the ac duct enters directly from the side where its located. We have a couple of other storage servers in dell desktop cases with the sides removed running above it. With direct cold air the fans are always on their slowest speed except a couple the warmer days during the winter when no ac is running, but it's still ~60*F ambient in the warehouse. This setup was done about 12 years ago, since then we have changed servers a couple of times and really could put everything back inside as the noise level is less than half of what it was back in the day. It is by no means a high end installation, really a temporary solution that has worked for over a decade.
  
  
  
  
  
  
  

 

[svtkobra7]

Apologies for the delayed response, I'm not sure why I don't always get an email notification of a reply ...

Great job.

• Many thanks (I actually wasn't expecting all the compliments LOL).

A minor suggestion is to use QuietRock instead of regular drywall. You could go further and use some of their other products (quiet seal, quiet putty) to further isolate noise.

• I researched sound abatement techniques (as related to construction) and thought that 5/8" sheetrock + Rockwool would be sufficient. (HD doesn't carry sound abatement drywall AFAIK LOL - and that was a PITA to transport here). I did use a product similar to quiet putty on the outlets though.
• If one were so inclined, you can take these techniques quite far, example second layer of drywall, floated, but not personally having experience with them I remained skeptical of their "bang for the buck" until I saw the effectiveness of the Rockwool ...

Not sure that the hum your girlfriend hears would have been avoided with additional insulation as that noise is created by the need to ventilate (fan, gap at bottom of door, etc.), but it might have helped. You could also use a Panasonic WhisperCeiling fan, for example. They are very quiet, and although as someone noted, not designed to run all of the time, they do have a 6 year warranty and the motors use ball bearings.

• It may be me hearing what I want to hear, but when it is completely silent and I have my head ~24" away behind the closet in the clothes' closet, if I can hear any fan noise, I'm nearly certain it is being reflected (?) off the hard wood floor and bouncing around the corner, i.e. I'm not hearing it through the wall (thanks to the sound abatement insulation - I think the stuff is amazing).
• I looked at the Panasonic model you mentioned during the planning phase, which was a good bit more expensive than the model I used. In the event, my "solution" was a flop, I wanted to write off as little as possible. Hindsight being 20/20, I absolutely would have sprung for the Panasonic knowing the solution works.

I appreciate your reply and hope that you enjoy the balance of your weekend.

 

[svtkobra7]

Agreed, and I like your solution of turning a 4-post upright. You made the most of a system that really isn't made to work that way.

Yeah, my numbers roughly match yours. Though I was expecting that fan to have a 3" duct. If the door is currently just over a floor, you could easily open up the gap to 1", and install a wood threshold at a later time.

• Definitely 4" ...
• 1" would result in more noise transference, but is it my current bottleneck? I think it is on the exhaust side.

That's an interesting point. By mounting the systems vertically, he's counting on the heat to rise naturally, and most of it will, but some may end up recirculating into the computer. If you can duct the exhaust fan to capture the heat straight from the computers, the room will be more efficiently cooled, and the computers will have a lower inlet temperature, which can in turn reduce noise.

• Recall the bit about the "right side up" chassis being 5.3C warmer than the "upside down" chassis? I think that precisely confirms your point about re-circulation. Not sure of how much is being re-circulated in an "upside down" orientation, but some for sure.

Another option would be to water cool the computers. That's more effort, but could potentially cut the noise to inaudible levels.

• Although it is becoming more commonplace in datacenters, it has typically been an approach to avoid due to the risk of damage from malfunction, i.e. it takes only a matter of seconds to replace a hot swappable fan if one dies (one could argue "best practices" have been thrown out the window here though).
• As to the effectiveness of water cooling, beyond the very edge case of immersing "sealed" HDDs in a liquid, I'm not aware of a "mainstream" approach to cool HDDs via water cooling (especially where there is absolutely no room between them). I always knew that HDD temps would be the battle fought, and not some other component overheating.

 

[svtkobra7]

A Supermicro fan too (I caught their "brand standard" colors before even looking for / finding their logo)! :

The server in our rack is actually sitting horizontal, about 4" was added onto the back of the enclosure so it would fit, the ac duct enters directly from the side where its located. We have a couple of other storage servers in dell desktop cases with the sides removed running above it. With direct cold air the fans are always on their slowest speed except a couple the warmer days during the winter when no ac is running, but it's still ~60*F ambient in the warehouse. This setup was done about 12 years ago, since then we have changed servers a couple of times and really could put everything back inside as the noise level is less than half of what it was back in the day. It is by no means a high end installation, really a temporary solution that has worked for over a decade.

I ABSOLUTELY LOVE THAT INSTALL - killer! Forced A/C + monitor (jealous) ... granted I use IPMI and there is no need to have a monitor + input devices in the closet, but it still would have been COOL (pun intended).

 

[Ole Slewfoot]

• Thanks for offering ideas for sound abatement (my fiance was mentioning tonight she always hears a "humm" - I don't hear it).
• Could you expand a bit on the baffle idea please, I can't picture it in my head?

• That bathroom fan doesn't actually exhaust externally. Rather to a ~600+ cubic foot "enclosure" below the concrete slab ceiling, above the closet ceiling, which is segregated from the rest of the "air space." (authoring a reply now in response to @Provincial that will explain more accurately).

I think this should make sense, the door closes against a couple baffles.they could be built freestanding like a really lopsided coffee table
Another thing to consider is fans mechanically transmit a lot of noise if they are hard coupled to a wall...If you use some rubber motor mounts, line the french cleat with something soft, or hang an inline fan from bunjie cords, it often helps a lot.

 

[madwi]

If you havent been there already, there is some really friendly and helpful people on the /r/homelab subreddit if you would like another avenue for advice/ideas.

 

[Provincial]

madwi, from your avatar, I thought you were going to suggest mounting everything with duct tape!

 

[svtkobra7]

Another thing to consider is fans mechanically transmit a lot of noise if they are hard coupled to a wall...If you use some rubber motor mounts, line the french cleat with somethi

ng soft, or hang an inline fan from bunjie cords, it often helps a lot.
[/QUOTE]

• Good call on using a dampener on the french cleat. I had never considered that ... and along those lines, that gives me an idea.
• Each V Rack has 9 holes for fasteners, not that all are needed and they are fastened to the plywood as follows ...
• T NUT HEAD [FLUSH] | PLY | SS WASHER * | V RACK | SS WASHER | PLASTIC WASHER | M5 RACK SCREW ...
• Perhaps adding a rubber grommet where the * is inserted could have a similar effect.
• I think a damper + MLV on the door may be the easiest, lowest hanging fruit, but being in between projects I'm inclined to be a little more aggressive.

I think this should make sense, the door closes against a couple baffles.they could be built freestanding like a really lopsided coffee table
Another thing to consider is fans mechanically transmit a lot of noise if they are hard coupled to a wall...If you use some rubber motor mounts, line the french cleat with something soft, or hang an inline fan from bunjie cords, it often helps a lot.

• You are going to think I'm on the extreme left side of the bell curve when I say this, but I still don't completely follow the concept after staring at it for 15 minutes trying to coax out the secret purpose.
• When I had thought of using a baffle previously, this is what I thought of:
  • install a baffle on the floor supported by the closet wall, or better yet,
  • construct it such that the top portion is affixed to the door and the bottom to the floor,
  • (see borrowed image, representing my thought)
  • and only when that door is closed do the two pieces come together to form the baffle
  • in either approach I'd suggest it be flush with the door.
  • The impact I thought it would have = deflect the sound waves as they attempt to escape the closet, such that the longer distance that must be traveled for them to ultimately escape results in a reduction to sound pressure level.
  • So I think you are proposing the same I construct, I just can't see it. (yet and my apologies for being dense)

In my markups, (on the left) the straight arrows represent air entering the closet and (on the right) those crooked arrows are the deflected sound waves. Do I understand, maybe?

 

[madwi]

madwi, from your avatar, I thought you were going to suggest mounting everything with duct tape!

Server rack rails are expensive! Duct tape a cradle and let them hang!

 

[Ole Slewfoot]

You got it.
Worst case scenario is a straight line form your ear to a fan blade.
Line of sight from fan inlet housing to you can be almost as bad.
I think my layout is better than that other pic in forcing at least 2 turns of at least 90 degrees(with a possible cost in airflow). If you hear wind noise from air rushing under the door, the inlet is undersized.

A quick construction method of my illustrated concept would be to make two 12" square end plates, and two 12" panels as long as the door is wide.
The lower panel is attached flush with the lower inside corner of the end caps, angled up toward the door at 45* (will end about 2" shy of the door)
The upper panel is flush to the top outer edge of the end plates, angled down at 45, and ends shy of the floor. Carpet or texture on the panels makes it more effective.

To operate, take the assembly, and set it in the closet so the door shuts against the open mouth.

Its possible you could get really unlucky and create a resonant tube, but that only happens to me with exhaust pipes on daily drivers.