Announcer: Welcome to not your father’s data center podcast brought to you by compass data centers we build for what’s next. Now here’s your host, Raymond Hawkins.
Raymond Hawkins: All right, thank you for joining us for another edition of not your father’s data center. I’m your host, Raymond Hawkins, chief revenue officer at compass data centers. I am joined by Uptime Institute, chief technology, officer Chris brown. Chris, thank you for joining us.
Chris Brown: Glad to be here.
Raymond Hawkins: And if I remember right Chris, aren’t you joining us from somewhere outside of Tulsa? Is that the best as close as we want to get?
Chris Brown: Yeah, I am just outside Tulsa, Oklahoma.
Raymond Hawkins: Chris, thank you so much for jumping on with us. We would love to hear a little bit about you. I know Oklahoma is home, but you did a stint down here when you went to school. Talk a little bit about your time in and out of Texas and in Oklahoma and how you… What path led you to end up being at Uptime Institute?
Chris Brown: Certainly, well, I’m actually from Texas north central, Texas, just north of Dallas. And I went to school at The University of Texas in Arlington and pursued a degree in electrical engineering. And it was kind of interesting how I got into the data center industry is I just fell backwards into it. I graduated from college in 1995. Data centers were, I mean, they were going strong, but they weren’t well widely known. The industry didn’t publicize itself very well while I was in college and in my senior year, trying to look for a job American airlines was at on campus and they approached me and wanted to talk about their sacred division. And so after a little bit of that, we kept the conversations going and that’s how I found myself in Oklahoma. So a Texas boy that swore up and down it that never live in Oklahoma has spent over the last 25 years here.
Raymond Hawkins: On the wrong side of the red river.
Chris Brown: On the wrong side.
Raymond Hawkins: All right.3
Chris Brown: It was a good thing because I got into data centers through Sabre and that life has been very good to me. I’ve been fortunate enough to be able to get into some different companies when they were in times when they were doing a lot of work and a lot of changes. So it allowed me to get 30 years worth of experience in about 10 or 15 years. And so it’s been very good and I’ve enjoyed working in the industry and look forward to many more years to come.
Raymond Hawkins: All right. So if I remember right, didn’t Sabre have a big presence up there outside of Tulsa, right? Wasn’t there a big data center facility in the Tulsa area? Am I remembering correctly?
Chris Brown: Yeah. Sabre had two data centers right on the airport property.
Raymond Hawkins: Okay.
Chris Brown: So if you think about it in Tulsa, Tulsa international airport, American Airlines had a long term lease on a lot of property just on the edge of the airport. And so when American Airlines built their data center, they built their first data center out of an old office building. And then they actually built underground data center there on the airport property that could continue to operate even if they had unfortunate crash there. And so they had two decent sized data centers there. Then when they spun off Sabre, Sabre built a third data center a little bit further off the airport, about 10 miles away, but still in that area. And it’s a fairly large data center and it’s still in use today. It’s about five acres under roof.
Raymond Hawkins: Oh my goodness. Five acres. That’s a chunk. All right. Okay. And at one point wasn’t Sabre part of American, and then it became its own outsourcing for lack of a better term businesses. Do I understand that correctly, Chris?
Chris Brown: Yeah. Sabre was created by American Airlines to handle their own reservations systems, which then expanded into their operations systems that basically every aspect of the airline was managed through the Sabre systems. They had tried for years and successfully in some areas to expand and provide those services to other companies and other airlines and rental car companies, hotels, and things of that nature. But they were seen as competitors. And so about 1997, they decided to spin it off into its own individual company. And so Sabre kept operating the data centers and providing the IT services and tried to expand into different companies in different areas.
Raymond Hawkins: So walk me through the transition from doing American and Sabre and travel and the data centers there in Tulsa. Where do you jump rails and head over into the Uptime Institute and start teaching the industry how to think about this stuff?
Chris Brown: Well, once again most opportunities are those that just are surprises rather than things that are planned. I worked with Sabre for a little while. Sabre was getting ready to spin off its data centers to ultimately EDS. But at the time we didn’t know who it was. And all I knew was Sabre was getting out of real property. And as an electrical engineer, real property is pretty much what I do. And so I left Sabre, went did a little stint in the petroleum industry with Citgo petroleum, working with their laboratories, their pipelines, their liberal plants, when they found out I knew something about data centers that kind of got into a lot of their laboratory stuff, but then EDS was operating those data centers, the old Sabre data centers. And I knew some folks that were there and they were having some troubles and I kept telling them how to fix the problems.
Chris Brown: And then one day a gentleman from Trammell Crow company who was their operating company called me up, and it was essentially put up or shut up, and it was an opportunity to go work for them. So I went, worked for them for a few years, did a little sideline as well after that. So Trammell Crow for about five years and then did some contract engineering work. Due to some family issues and I needed to be home more often because as you probably know, if you’re working in a data center, it’s pretty much an 80 hour a week gig or more. And then those family issues got solved. And I was looking to get back in with the team and a friend of mine that I’d known for years and worked within Sabre and some other places called me up because the Uptime Institute needed some engineering help. And so I took that opportunity and started working with Uptime Institute. And so with Uptime Institute, I started that in 2010 and have worked my way up from consultant, delivering their certifications for data centers, to being the CTO of the company.
Raymond Hawkins: Be 12 years here next month or two months, won’t it? Is that about right?
Chris Brown: It will be. I started in January, 2010.
Raymond Hawkins: All right. 12 years. That’s a long gig for anywhere. So congratulations on that. All right. So talk to us. So we’ve gotten you a Texas boy that ends up on the wrong side of the red river. Got schooled on both sides, right? UT Arlington and I think Oklahoma State, right?
Chris Brown: That’s correct.
Raymond Hawkins: So you get edumacated on both sides of the red river and understand the American Airlines piece of it. So now you’re at the Uptime Institute. Tell me what in the early days. Because if I think about it, I know you’re talking about data centers in 95 and 96. I mean there have been computers living in buildings for a long time, but as far as the wholesale sort of commercial approach, that’s an early two thousands kind of a thing. What did the world look like 5, 6, 7 years in when you joined the Uptime Institute and your commercial wholesale data centers were still fairly new in 2010? What were you guys doing? What were you guys teaching the industry? Talk to me about the early days and love to hear how it’s transformed in your 12 years.
Chris Brown: Well, back then when the retail data center industry was just getting going and establishing its footprint, they were needing to capture business from the enterprise and convince the enterprise companies that it was better to use them for their data center space rather than build their own and operate their own data center space. And one of the challenges that was happening at the time was having all those folks that had spent 10 years, 15 years, 20 years designing and building data centers for specific companies and operating them for specific companies to trust anyone outside of them has a place to house the IT. So that was a big challenge then. And we worked with some of the co-location providers at the time and talking to them about the tier standards, because there were some that were not really adhering to really any standards at all. And they were trying to learn the ropes as well. It was a burgeoning industry. And so we started talking to them about the tier standards and tier certification and things to prove that they were designed, but also were compliant and performed to those tier standards.
Raymond Hawkins: So Chris did the concept of tier, is that an Uptime concept? Is that where tier started?
Chris Brown: Yes.
Raymond Hawkins: Or was there already a… Okay, so putting that title, you’re a tier two tier three tier four data center started at Uptime.
Chris Brown: It did, it started at Uptime Institute and it was one of those things that, uptown Institute had been talking about tier since before 2006. And there was a lot of discussion there and Uptime Institute had been involved in the original, I guess you say, birthing of tiers. Well before that, but they were starting to commercialize it, get the market and the industry familiar with the tier standards, tier requirements, things of that nature. And so when I came on board with the Uptime Institute, I’d had exposure with tier before then. And when I came out at the Uptime Institute, we were trying to… We talked to and convinced a lot of players in the co-location industry that the way to communicate to enterprises, that their facilities were quality, that they were designed and built to the rigorous standards was to use tier certifications for that.
Chris Brown: And so that was what was going on at the time with co-location data centers and things have changed so much since I first started in data centers, since first started at Uptime Institute. When I started at Uptime Institute, there were four people delivering the technical work and they were all US based. Today we have 34 engineers scattered across 13 different countries and-
Raymond Hawkins: Oh, wow.
Chris Brown: … It’s still growing. So it’s definitely changed a lot, but the data center industry has changed a lot. When I first started chilled water plants were the norm because the electrical power that was required to run a chill water plant was about 25% of what it would be at with direct expansion. And so the chill water plants were all customized. The people that are operating them were well trained and had to be well trained and were highly experienced.
Chris Brown: When I got into the industry most of the mechanics and the chiller operators, they were in their late 40s. And had been doing it forever. In fact, when I was responsible for operating data centers, one of the things that I did was stole people from the hospitals, because they knew understood mission critical and hospitals had the same systems as data centers. But over the years, what we’ve seen is as technology has improved the direct expansion technology, as well as using evaporative cooling and other approaches has brought the cost of using direct expansion down to pretty close to what a chill water plant can run. And so then that little elevated cost of direct expansion in terms of energy costs can be offset because the rigorous skillsets of the operators are not quite as much.
Chris Brown: If you’re having to worry about running chillers, you got to worry about flows, hydraulics, pressure differentials, temperature differentials, direct expansion. Most of the time you tell it what step points you want it to be. You tell it to run, it runs. And if it doesn’t run well, then you have to have a technician anyway, because you’re dealing with refrigerant gases and other things that requires special licensing to deal with. So that’s been a major change that we’ve seen in the industry in the last 10 or 15 years is just the moved to even to large scale direct expansion plants.
Raymond Hawkins: So before we get too far down the technological changes, especially since you were there when tiers became a thing. Let’s go back. It’s funny a little bit like Coca-Cola or Kleenex, everybody just says Kleenex and what they mean is tissue paper. When people say tiers, I mean, that’s an Uptime thing, but everybody knows, I see it in RFPs. I see it in any communication with brokers. They say, “Hey, here’s the tier.” Will you walk us through? I don’t even know if I can even begin to say what a tier one data center is just quick bullets, tier one, tier two, tier three, tier four. What are the major differences?
Chris Brown: The tier rating system has four tier levels. One through four they’re progressive, which means that the subsequent tier levels have all the requirements of the previous tier levels. And then add some on top of it. The first three tier levels are about providing increasing levels of opportunity for maintenance without impacting a critical load. So if you start with tier one, a tier one data center has just enough of everything. So just enough in onsite power production which is typically engine generators, just enough cooling capacity, just enough UPS capacity for whatever load that the data center’s going to operate. So you could have a single UPS, a single chiller, a single engine generator and that’s tier one. Tier one provides no opportunities for maintenance without impacting a critical load.
Chris Brown: Then you step up tier two, which is redundant capacity components. So if you think of data center, all systems have capacity components. So chiller is a capacity component. It’s something that creates the capacity, the piping and the pumps are a distribution pass. All right, same thing in electrical, UPS system itself is a capacity component, but all of the cabling, the PDUs, things of that nature or distribution pass. Tier two requires at least one redundant capacity component for all systems. So you’d have to have at least one redundant chiller, one redundant air handler, one redundant UPS. It does still allow a single distribution path. So where tier two steps up into is, you have the opportunity to provide, to conduct maintenance on your capacity components without impacting your critical load. But the critical distribution path is still a single path and doing any work on the critical distribution path will still impact your critical load.
Chris Brown: So then you get to tier three, which has all the requirements of tier two. So you have redundant capacity components, but tier two only has a single distribution path. Tier three requires redundant distribution paths. So tier three gets into full concurrent maintainability. So every capacity component, every distribution path, every system touching a critical system must be able to be isolated and for planned activities. And those planned activities could be maintenance, upgrade or replacement. And it has to still be isolated without impacting a critical load.
Raymond Hawkins: We have redundant capacity as well as redundant paths. So I can do all the work without changing delivery to my critical systems is why I think the key designation for tier three. So hit us with tier four Chris.
Chris Brown: Okay. And tier four. So tier one through three is about increasing opportunities for maintenance, from no opportunity for concurrent maintenance, without impacting critical load, which is tier one all the way through tier three, which is full maintenance without impacting your critical load. Tier four adds the idea of fault tolerance. So if you think about it, tier three is about planned activities. You plan to go perform maintenance or do an upgrade on a system. Well, tier four adds in the idea of an unplanned activity, which is a fault or a failure. So with tier four, the requirement basically is that all systems must be able to respond to a fault or failure without operator intervention and without impacting critical load.
Chris Brown: So if you think about it, if you’re running a chill water plant and you lose a chiller and the chiller spins offline, the system has to detect that the chiller has been lost, start the redundant chiller up and continue to serve the load. And it’s without operator invention, it’s all automatic autonomous response. And so tier four gets into being able to deal with any single event, whether it be if a planned activity or an unplanned activity.
Raymond Hawkins: Got it. All right. And there’s the four that you hear thrown around pretty casually in our industry today, which is pretty incredible that you were there in the early days. I mean, tier four is, tier three, pretty casually mentioned standards that everyone largely has their arms around, or at least thinks they do. So I appreciate the refresher. All right. So the business has grown incredibly. You’re in, I think you said 13 or 14 countries. Talk to me a little bit about as the industry, and I liked at the beginning, Chris you talked about, we had to convince enterprises that they could put their computer somewhere else without a doubt had to convince them that it could be run safely, securely, reliably in another facility. What is it that Uptime is helping providers and customers with today? I certainly get those early days of, I’m not sure I’m ever going to let these servers out of my building. That question seems to have been answered. What challenges are you guys answering for people today?
Chris Brown: Well, we still answer some of those same challenges. A lot of our tier certification work is about helping people to ensure that their facilities are designed to provide the availability and resiliency that they need. If you think about a lot of companies, a lot of companies don’t hire their own engineering staff. When I worked at American Airlines, I was on a team of engineers that the company actually employed, and a lot of companies just don’t employ those detailed design engineers anymore. So one of the things that we do is we help companies that are building data centers to ensure that they’re getting what they paid for when they engage with design engineers or construction companies. So we ensure that their designs are tier compliant. We ensure that the facility was built to the design, but also performs to it and meets all the tier requirements.
Chris Brown: We help clients with understanding how to operate their data centers, because as you know, designing a data center to the highest standards only gives you the opportunity to meet your availability needs. But you have to… The operations is where that investment is realized over time. So you invest in a large facility with high quality systems and well at a good design, but if you don’t have a good operations team, you’re not going to realize the availability that facility can give you. So we help them with that as well. We’re also helping clients understand even non-tier rated facilities, existing facilities. We help them understand what their risks are. So we help them to look at their operations teams, their actual facilities, and help them understand what that facility can give to them over the long haul, how it’s going to help them meet their specific business needs, but also what those risks are and where they need to plug some holes of those risks. So that’s where we’re at today.
Raymond Hawkins: Got you. Good stuff. So, Chris, as you guys look at the industry and you got a great view on where our space is going, I’d love it if you’d tell me, “Hey, Raymond, one is, no one’s talking about this, but we ought to be so with that, no one’s talking about it but we ought to be.” What would fit in that category for you? No one industry’s really talking about it, but I think this is going to be something we’re thinking about in the future. I’m hitting you with that one by a little bit of surprise. And then everyone’s talking about this, and I don’t think it’s as big a deal as it is, as everyone’s making it out to be. Both of those two categories.
Chris Brown: One that no one’s that few people are talking about that I think they need to be talking about… Kind of goes hand in hand with the example I would have for both of your questions. They go together. So sustainability is a big industry topic today. And everybody’s looking at sustainability and countries are looking at how to get their grids more sustainable and how to advance their grids to have more renewables. We have, and data centers are doing the same sort of thing. In that though we’re going to be in periods where there’s going to be a little bit of instability in the grids. Texas saw that this year. I mean, they had some problems because we had what I call the great freeze. So cold temperatures that no one had seen in a long time, no one expected even up here in Oklahoma where we had 19 degrees below zero is something we hadn’t seen in a long time. And we see some Culver temperatures than Texas does.
Chris Brown: But it created some moments of instability in the grid, because of course there was less power available than there was load. And so one of the things that, everybody’s talking about sustainability and it’s a big deal. And along with sustainability, you have engine generators. And everybody looks at the engine generators from the data center world as being big polluters. And the reason they look them as big polluters is when they operate sure enough, they do pollute into the air. But of course, there’s processes to help reduce that such as urea treatments, catalytic converters, those sorts of things. But most people don’t realize that they only pollute when they run and they don’t run that often, but they’re a good insurance policy for the business.
Chris Brown: Because when you need them, you need. And if you didn’t have them, when you needed them, you’d be wishing you did. But one of the things that, what we’re seeing and what I’m thinking, and only a few people are talking about this is that we can’t move our energy grids in the US or any other country, any modern country from where we stand today, straight to a green carbon neutral resilient grid, there’s going to be problems along the way. I mean, we understand how to operate with coal fire plants and with nuclear plants and with natural gas fire plants, because all that’s base load generation, renewable is not base load generation. So over time, we’re going to try some things and we’re going to have some successes and we’re going to have some failures. And, so one of the things that I think is very, that the industry needs to be at and sort of touting, if you will.
Chris Brown: And we need to be ready to do is when there’s instability in the grids, because there’s not enough capacity for the load, data centers have a huge, they consume a huge amount of power. We all know that in a very small footprint, but we have our own systems for generating power on site when we need to for emergency situations. And I think that data centers can serve a big role in the sustainability world, helping us figure out how to get the right… What is that right balance between base load generation and renewable sources on the grid. And how’s that look? What are the control systems need? How do they need to change? How those algorithms need to change? Because when you stub your toe and there’s a time when for some reason the power companies didn’t anticipate a heat wave like you would have in Texas or a sudden cold snap, which is going to put a lot more load on the grid.
Chris Brown: Data centers can help out, because they can pull huge amounts of load off of the grid and help to stabilize the grid. But they can only do that with engine generators and other reliable sources of power that we serve today. And so I think that’s a big piece that we’re going to be spending a lot of time… The data center industry can help that out. So I think the thing that people are talking about is sustainability. What they’re not talking about is how we can help, not just our industry, but the larger society as a whole get to where we want to be.
Raymond Hawkins: Yeah. I want to make sure I understand what you’re saying. What I think I hear you saying is, “Hey Raymond, everyone talks about sustainability and yes that’s a thing.”
Raymond Hawkins: And yes, they look at the data centers industry and say, “Wow, you guys eat up a lot of juice.” So let’s think about the juice for a state as one set of capacity.
Raymond Hawkins: What I think I hear you saying is, “Hey, when a system is strained, when a state of Texas’s capacity is strained, all the data centers conceivably could go offline and run on their generators and produce their electricity on site, thus providing relief to the grid and wins the right time to do that. And how do you do it? And how do you compensate him for that? How do you think about that?” Is that what I hear you saying?
Chris Brown: That’s exactly what I’m saying. The stint that I had with the petroleum ministry at Citgo petroleum. Citgo is a company that has roots years and years in the past. And they have a lot of power contracts that go back to fifties and the forties and things of that nature. And they had set asides with the power companies because they run huge pumps. 2000 horsepower pump motors, all powered by electrical power. And they could get at their facilities called by the power company and said, “Hey, look, we need this much load pulled off the grid for this amount of time.” Now if you’re just pushing power to pipelines, you’re just pushing that product on the pump lines. Well, if you have to wait an hour that’s okay. So they’d pull the power off the grid that would free up capacity. And then the power companies would then sell that on the spot market, which the difference between what Citgo was paying per KWH at their contract and what they would get on the spot market was a lot, there’s a big Delta there.
Raymond Hawkins: Sure.
Chris Brown: And Citgo would get a-
Raymond Hawkins: Because it was a high demand window.
Chris Brown: Yeah. And Citgo would get a small cut of that action. And so what did a couple things, they were good corporate citizens, so they could help free up power when power was really required. And they got a little bit of reward from it, from the power company, because they got a little cut of the profits off of that. And so what I’m saying is that… And it’s already happening in some places where data centers can sign up with power companies to be part of that set aside. And if a power company sees, “Hey, look, my grid’s becoming unstable because I don’t have enough capacity and it’s going to take me a while to spin up some more turbines or something.” Then they can call up the companies that are on that list. And they’ll pull that load off of the grid.
Chris Brown: Well, that helps to stabilize the grid because you’ve now starting to match capacity to load and give them time to respond with some of their base load generation. And so I think that’s an approach that we need to be looking at as an industry, partnering with the utility companies. Because as the utility companies go to modernize the grid and transition how we’re producing power, we’re going to have times where we’re going to stub our toes. Right. I mean, living in Texas, I remember those times where you’d wake up in the morning and it would be, 50 degrees in the morning, but by the afternoon it was in the eighties or nineties and everybody comes home and throws on their air conditioners because they didn’t have them on previously. And you’d get times of brown outs. And that was with years and years of experience of baseload generation with coal fire and natural gas fire power plants. Now let’s add some wind into it. Let’s add some solar into it. And you’ve got a whole different system now.
Raymond Hawkins: Yeah. And I think that’s one of the… And we’re not going to get lost in electrical engineering, but I do think that’s one of the things that people struggle to understand is that storing electricity is hard. You generate it when you need it largely. I’m oversimplifying. And you can tell a coal fire, power planter, a nuclear fire, power planter, hydro plant, you can tell it to run or not to run. You can’t tell the sun to shine and you can’t tell the wind to blow. It blows when it wants. And it shines when it wants. So you don’t have the on, off switch ability of generation that you have from traditional power facilities. So it’s like you said, when the demand comes, if you don’t have the supply, we’re breaking the system.
Chris Brown: Well, and the other part of the problem is because the wind cannot blow or you can get heavy overcast all of a sudden, so you can lose some of your capacity from your renewable systems. And it takes time to bring those large turbines online, get them spun up, producing power. If we think in our data centers, you can have a quick start engine generator online producing power in 10 seconds. Not so much at the power company level.
Raymond Hawkins: Right. The big grid gets the longer it takes. No question. Very cool stuff. Well, Chris, this has been awesome. I really appreciate you spending a little time with us and hanging out and talking. Not only are we good friends with the Uptime Institute as an institution, but also with lots of folks there. And we’re grateful to be partners with you guys and grateful to have you come, tell us a little bit about your story as well as help people understand what Uptime does in the space and how you guys are really, I think shepherding the industry and advising people on how to, I like the way you said it, “Hey, you get, you paid for this design. You paid for this deployment. Is it actually there? And does it actually work?”
Raymond Hawkins: Because at the end of the day, all people want to know is, “Hey, are my servers on? And can I talk to them.” And you guys help folks do that? And we appreciate it and appreciate the standard you guys set in our industry.
Chris Brown: It’s been our pleasure.
Raymond Hawkins: Chris. Thank you so much.
