Hi Hubr, I am Taras Chirkov, director of Linxdatacenter data center in St. Petersburg. Today in our blog I am going to tell you what role cleanliness plays in modern datacenter and how to measure it, how to achieve and maintain it.
One day we were approached by a client in a data center in St. Petersburg about a layer of dust on the bottom of the equipment rack. This was the starting point of the investigation, the first hypotheses of which suggested the following :
- dust enters the server rooms from the soles of the shoes of data center employees and customers,
- is brought in through the ventilation system,
Blue shoe covers to the dustbin of history
We started with shoes. At that time the problem of cleanliness was solved in the traditional way: a container with shoe covers at the entrance. The efficiency of this approach did not reach the desired level: it was hard to control their use by guests of the data center, and the format itself was inconvenient. We quickly gave up in favor of more advanced technology in the form of a shoe machine. The first model we installed was a failure: the machine too often tore the shoe covers when trying to put them on the shoes, its use was more irritating than easy.
The experience of our colleagues in Warsaw and Moscow did not solve the problem, and finally we opted for the technology of thermal cladding on shoes. With the help of the thermal film "shoe covers" can be put on shoes with any sole – right up to a thin women’s heel. Yes, the film also sometimes slips off, but much less often than the classic blue shoe covers, and the technology – many times more convenient for the visitor and more modern. Another important (for me) plus – the film easily covers the largest sizes of shoes, in contrast to the traditional booties, tearing when you try to put them on a size 45. To make the process even more modern, we installed urns with automated opening of the lid by motion sensor.
This process looks like this :
Guests immediately appreciated the innovation.
Dust in the wind
Having cleaned up the most obvious channel of possible space contamination, we have moved on to more subtle matters: the air. It is quite probable that a large part of dust gets into the server rooms through ventilation due to insufficient filtration, or is brought in from the street. Or is it the poor quality of cleaning? The investigation continued.
We decided to take measurements of the particle content in the air inside the data center and invited a lab that specializes in monitoring air quality in clean rooms for special purposes to do this work.
Laboratory staff measured the number of control points (20), made a schedule of sampling to track the dynamics and create the most accurate picture. The cost of the entire process of measurements by the laboratory was about 1 million rubles, which seemed to us completely impractical, but gave a number of ideas for its own implementation. Along the way it became clear that the lab – it’s good, but the tests should be carried out in the dynamics and to constantly refer to their services is extremely inconvenient.
After looking at the lab’s planned activities, we decided to look at more utilitarian devices for do-it-yourself work. As a result, we managed to find the necessary tool for this task – an air quality analyzer. Here’s one like this :
The device shows the content of particles of different diameters (in micrometers).
Revisiting the standards
This instrument analyzes the particle count, temperature, humidity and shows the results in units of measurement according to ISO standards for that parameter. The display shows the levels of particles of different diameters in an air sample.
At the same time we blamed the filters: at that time we used G4 filter model inside the server rooms. This model provides coarse air purification, so there was a chance of letting in particles which led to contamination. We decided to try out the F5 fine filters, which are used in the air conditioning and ventilation systems as second stage filters (afterfilters).
The investigation has been carried out – we can proceed to control measurements. As a benchmark we decided to use the requirements of ISO 14644-1 standard for the number of suspended particles.
Classification of clean rooms by number of suspended particles.
It would seem – measure and compare according to the table. But it is not that simple: in practice it was quite difficult to find standards for air purity in the server rooms of data centers. It is not explicitly prescribed anywhere, not a single organization or industry institute. And only on the Uptime Inside Track internal forum (access to it is given to those who have been trained according to Uptime Institute programs) there was a separate discussion on the subject. As a result of studying it, they leaned towards ISO 8 – the penultimate standard in the classification.
The first measurements showed that we underestimated ourselves – the indoor air test results gave us ISO 5 indoor air compliance, which was well above what Uptime Inside Track members wanted. And by a wide margin. We are a data center, not a biological laboratory, of course, but for the particle concentration in the air to be equal to ISO 8 – it has to be a facility of the "cement factory" class at least. And how you can apply the same standard to the data center – not very clear. At the same time, we got the result at ISO 5, making measurements at air filtration with G4 filters. That is, the dust cannot get into the racks through the air, F5 filters were too much, and they were not even used.
A negative result is also a result: we have continued to look for the cause of the pollution in other directions, and air quality control has been included in the quarterly checks, combined with BMS sensor checks by certified devices (requirements of ISO 9000 and customer audits).
Below is an example of the report, which is filled out based on the data obtained during measurement. For greater accuracy, measurements are made with two devices – the Testo 610 and the BMC sensor. In the table header there are limit values for the devices. The deviation of the indicated parameters is automatically highlighted in color for easy identification of problem areas or time periods.
We are clear : the difference in instrument readings is minimal and the particle concentrations are well below the limit.
Through the back door
Since there were other entrances to the clean rooms besides the main customer entrance, where we installed a shoe machine, there was still a need to prevent dirt from entering the data center through them.
It is inconvenient to put on/remove shoe covers during procedures of unloading equipment, so we found a machine for cleaning soles. Convenient, functional, but affected by the human factor in the form of non-obligatory approach to this machine. Basically, the same as the shoe shine machine at the main entrance.
To solve the problem, they started looking for cleaning options that could not be avoided : sticky carpets with separable layers were the best way to do this. In the process of authorization at the entrance doors, the visitor necessarily stands on such a mat, removing excess dust from the soles of his shoes.
The cleaners tear off the top layer of such a mat every day, there are a total of 60 layers – enough for about 2 months.
When I visited Ericsson’s data center in Stockholm, among other things, I noticed how they solve these issues there: along with tear-off layers, Sweden uses reusable Dycem antibacterial carpets. I liked the idea because of the reusability principle and the ability to provide a large coverage area.
Magic antibacterial carpet. Too bad it’s not an airplane, and it could have been – at that price!
We had a hard time finding representatives in Russia and evaluated the cost of the solution for our data center. We ended up with a figure almost 100 times more expensive than the solution with multilayer carpets – about the same 1 million rubles as the project with air purity measurements. In addition, we found the need to use special cleaning products, of course, available only from this manufacturer. The solution also fell away by itself, we settled on a multilayer option.
I would especially like to point out that all of the above measures did not cancel the use of janitorial labor. In preparation for the certification of Linxdatacenter data center in accordance with Uptime Institute Management Operations standard, it was required to clearly regulate the actions of cleaning staff in the data center. Detailed instructions were drawn up prescribing where, what and how they should do.
A couple of excerpts from the manual :
As you can see, everything is prescribed, literally every aspect of work in a particular room, allowable cleaning agents, materials, etc. No detail, even the smallest, is left unattended. Instruction signed by each employee of the service. Server rooms, electrical rooms, etc. are cleaned only in the presence of authorized employees of the data center, such as the engineer on duty.
But that’s not all
Also on the list of measures to ensure cleanliness in the data center: rounds with visual inspection of the premises, including weekly inspections of racks to detect wire clippings left inside, remnants of packaging from equipment and components. For each such episode, an incident is logged and the customer is notified to correct the irregularities as soon as possible.
Also, for the unpacking and setting up of equipment we created a separate room – this is also part of the cleaning policy of the company.
Another measure we took from Ericsson is to maintain a constant airflow in the server rooms: the pressure inside the rooms is higher than outside, so there is no draft inside – we will talk about this solution in detail in a separate article.
Finally, we got ourselves robot assistants for rooms that are excluded from the list of rooms available for visits by cleaning personnel.
The grille on top not only gives +10 to the robot’s protection, but also allows it to avoid getting stuck under the vertical cable trays of the racks.
Unexpected finding as a conclusion
Cleanliness in the data center is important for the operation of server and networking equipment, which pulls air through itself. Exceeding dust tolerance will result in dust accumulation on components and a cumulative temperature increase of up to 1 degree Celsius. Dust reduces cooling efficiency, which can result in significant indirect costs per year, as well as affecting the resilience of the facility as a whole.
We may say that it is a speculative assumption but Uptime Institute experts who certificated Linxdatacenter data center according to Management Operations standards pay the most serious attention to cleanliness. And we were very glad to get a very favorable assessment: our data center in St. Petersburg exceeds the certification requirements. The expert of the institute called us "the cleanest data center he has seen", moreover, our data center is used by Uptime as an example of solution to the problem of cleanness of the server rooms. Also, we easily pass any client audit on this parameter – the most serious requirements of the most capricious clients are satisfied with exceeding.
Let’s go back to the beginning of the story. Where did the contamination on that very complaint from the beginning of the article come from? The part of the customer rack that triggered the whole "clean in the data center" project was contaminated from the moment the rack was brought in and installed in the data center. The client had not cleaned the rack by the time they brought it into the server room – when checking the neighboring racks installed at the same time, it turned out to be the same situation with dust there. This situation prompted the addition of a cleaning control item to the customer’s rack installation checklist. The likelihood of such things should also never be forgotten = forewarned is forearmed. That’s all about "cleanliness and dictatorship" in our data center, I’ll tell you about pressure sensors in the next article, but for now, ask questions in the comments.