For secure air-gapped AI training, the essential network settings are a physically isolated LAN with no route to the internet, strict firewall rules that only allow necessary internal traffic, and disabled USB/Bluetooth; the essential GPU settings are enabling ECC memory, locking the power/clock limits to a stable value, and disabling any remote management or telemetry features. This article explains the key configuration choices for both network and GPU that keep a training cluster both secure and high-performing.
Why Air-Gapped AI Training Needs Special Settings
An air-gapped system is physically isolated from the internet, which is the strongest defense against remote attacks. However, even in an isolated environment, you still need to manage data flow, prevent accidental leaks, and ensure that the hardware runs reliably for long training jobs. The settings you choose affect both security (who can access the system) and stability (whether a 3-day training run completes without crashing).
Network Settings for an Air-Gapped Cluster
When you set up a network for air-gapped training, you are building a private LAN that has no connection to the outside world. Here are the core settings to configure:
- Use a dedicated switch/router: Connect all nodes to a switch that has no uplink to the internet. Do not connect this switch to a corporate network unless you have a firewall that blocks all outbound traffic.
- Assign static IP addresses: Use a private subnet (e.g., 10.0.0.0/24) and assign static IPs to every node. This avoids DHCP failures and makes firewall rules more predictable.
- Disable DNS and DHCP if possible: If you only have a few nodes, you can use /etc/hosts files instead of a DNS server. This reduces the attack surface.
- Firewall rules: Allow only the ports needed for training: SSH (22), distributed training communication (e.g., NCCL uses 6000-6100, but you can set a custom range), and possibly a shared storage protocol like NFS (2049) or SMB (445). Block everything else.
- Disable IPv6 if not needed: IPv6 can sometimes auto-configure and create unintended routes. If you are not using it, disable it on all interfaces.
- No default gateway: Set an empty gateway on all nodes. This ensures that even if a packet tries to leave, it has no route.
GPU Settings for Stability and Security
Modern GPUs have several settings that are often left at default but matter in an air-gapped environment. The most important are:
- Enable ECC memory (if available): On NVIDIA A100, H100, and similar data center GPUs, ECC (Error Correcting Code) protects against memory bit-flips. This is critical for long training runs where a single error can corrupt the model. It is usually enabled by default, but verify with
nvidia-smi -q. - Lock power and clock limits: Use
nvidia-smi -plto set a power limit (e.g., 200W) andnvidia-smi -lgcto lock the GPU clock. This prevents the GPU from boosting to unstable levels and reduces heat, which increases reliability. - Disable GPU telemetry and remote management: Some GPUs have out-of-band management (like NVIDIA’s Baseboard Management Controller) that can be accessed over the network. Disable these if you do not need them, or at least change default passwords.
- Use the NVIDIA datacenter driver: The standard gaming driver is not optimized for training and may include unnecessary features. The datacenter driver is designed for headless servers and has better stability.
How to Choose Between Security and Performance
There is a trade-off: stricter security often means slightly lower performance. For example, enabling ECC can reduce memory bandwidth by a few percent, but it prevents silent data corruption. Similarly, locking clocks to a lower value reduces peak performance but ensures consistent training times. For most AI training, stability and correctness are more important than a 5% speed boost. You should also consider the threat model: if you are training on sensitive data, a small performance hit is worth the security.
Common Problems and How to Avoid Them
- Accidental internet connection: A single node with Wi-Fi or a USB modem can break the air gap. Physically remove all wireless adapters and disable any unused network ports in the BIOS.
- NCCL timeouts: If you set firewall rules too restrictively, distributed training may fail with NCCL timeouts. Make sure the ports for NCCL are open between all nodes, not just from the master.
- GPU thermal throttling: In a closed rack, GPUs can overheat. Set a power limit that keeps temperatures below 80°C. Use
nvidia-smi -q -d TEMPERATUREto monitor. - Clock drift: If you are using distributed training, all nodes must have synchronized clocks. Set up an NTP server inside the air-gapped network (pointing to a local time source) to avoid drift.
Comparing Air-Gapped vs. Cloud Training
Air-gapped training is the most secure option because there is no network attack surface. However, it requires more manual setup and maintenance. Cloud training (e.g., on AWS or Azure) is easier to scale but you must trust the provider and use encryption. For highly sensitive data (e.g., medical records, proprietary models), air-gapped is often the only acceptable choice. For less sensitive work, cloud may be more cost-effective.
Cost and Hardware Considerations
Air-gapped training requires you to purchase and maintain all hardware yourself, which can be a significant upfront cost. You also need to consider the cost of electricity, cooling, and physical security (e.g., a locked server room). However, you avoid ongoing cloud fees. If you are starting small, you can use a single workstation with one or two GPUs, but for large models, you will need a cluster of servers with multiple GPUs each.
Safety and Physical Security
An air gap is only as strong as the physical security around it. Ensure that only authorized personnel can access the server room. Use BIOS passwords and full-disk encryption to protect data at rest. Also, disable USB ports (or physically glue them shut) to prevent someone from plugging in a USB drive that could introduce malware or exfiltrate data.
Alternatives to a Full Air Gap
If a full air gap is not feasible, you can use a “logical air gap” with a firewall that blocks all outbound traffic and only allows specific IPs. Another option is to use a separate VLAN for training and a jump host for access. These are less secure but easier to manage. For most organizations, a logical air gap is a reasonable compromise.
Frequently Asked Questions
Do I need to disable ECC for faster training?
No. ECC has a small performance impact (1-3% on memory bandwidth), but the risk of silent data corruption is far higher. For training, always keep ECC enabled.
What ports do I need to open for distributed training?
Typically, SSH (22) for management, and a range for NCCL (e.g., 6000-6100). If you use a framework like PyTorch, it may use additional ports for the master node. Check the documentation and open only those ports.
Can I use Wi-Fi in an air-gapped environment?
No. Wi-Fi is a security risk because it can be sniffed or used to bridge to other networks. Use only wired Ethernet connections.
How do I update the GPU driver in an air-gapped environment?
Download the driver on an internet-connected machine, transfer it via a secure USB drive (after scanning for malware), and install it locally. Do not connect the air-gapped machine to the internet for updates.
What is the best power limit for a GPU?
It depends on your cooling and workload. Start with the default, then monitor temperatures. If the GPU is above 80°C, reduce the power limit until it stays below that. For many data center GPUs, 70-80% of the maximum is a good balance.
Related
General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.
More in this bed
- Install ChromeOS: Convert Laptop to Chromebook
- Fix OneNote App Not Working on Android
- Move Games to SSD Without Reinstalling
- Move Windows 11 Taskbar to Top or Side: Easy Steps
- Windows 11 Calendar Wrong Day? Fix It Here
- How to Use curl Command in Linux