Skip to main content

Compute Node Hardware Profile Documentation

Document Version: 1.0 Date: October 26, 2023 Scope: Detailed hardware and software specifications for the local compute cluster nodes supporting AI inference and control plane operations.

🚀 Overview and Architecture Notes​

This document provides a comprehensive technical profile of the three primary compute nodes within the local infrastructure. These nodes are responsible for executing various services, including large language model (LLM) inference, data ingestion, and system orchestration.

Network Topology​

All compute nodes utilize Tailscale for secure, private network connectivity. This ensures that internal communication remains isolated from public internet exposure.

ComponentProtocol/ServicePurpose
TailscaleWireGuard VPNSecure L3 networking and IP assignment across all nodes.
Control Plane AccessSSH / API CallsRemote management and service interaction (primarily from the Control VPS).

Software Stack Summary​

The compute environment operates on a mixed OS architecture, optimized for specific hardware capabilities:

  • AI Inference: LM Studio v0.4.x is deployed across the local nodes to manage model loading and inference execution. The underlying acceleration framework varies significantly (NVIDIA CUDA vs. Apple Metal).
  • Control/Orchestration: The Control VPS runs a dedicated set of services for system management, scheduling, and data persistence.

💻 Node Profiles​

1. Compute Node: r7-5700x (GPU Inference Workhorse)​

This node is optimized for high-throughput GPU inference using NVIDIA CUDA acceleration. It serves as the primary dedicated compute resource for LLM tasks requiring significant VRAM and parallel processing power.

Hardware Specifications​

ComponentModel/SpecificationDetailsNotes
CPUAMD Ryzen 7 5700X8 Cores / 16 ThreadsExcellent multi-threaded performance for pre/post-processing tasks.
RAM32 GB DDR4Standard ECC or non-ECC memory.Sufficient system RAM for OS and model context management.
GPUNVIDIA GeForce RTX 3060 Ti8GB GDDR6 VRAMDedicated accelerator for LLM inference via CUDA.
GPU Bandwidth448 GB/sMemory bandwidth specification.Critical metric for data transfer efficiency during inference.
Operating SystemWindows 11 ProDesktop OS environment.Required for specific driver and software compatibility (e.g., NVIDIA drivers).

Network & Software Details​

  • Tailscale IP: 100.72.117.120
  • Primary Role: High-performance LLM Inference.
  • Key Software: LM Studio v0.4.x (Utilizes NVIDIA CUDA backend).

Operational Notes​

  • The Windows environment necessitates careful management of driver updates and resource allocation to ensure stable performance for the GPU workload.
  • Performance is highly dependent on the efficient utilization of the 8GB VRAM capacity. Larger models may require quantization or offloading techniques.

2. Compute Node: anguss-mac-mini (Apple Silicon Inference)​

This node leverages Apple's unified memory architecture and Metal framework, providing a unique inference capability optimized for efficiency and low power consumption.

Hardware Specifications​

ComponentModel/SpecificationDetailsNotes
CPU/GPUApple M4 ChipSystem-on-a-Chip (SoC)Unified architecture combining CPU, GPU, and Neural Engine.
Memory16GB Unified MemoryShared memory pool across all components.High bandwidth is key; capacity limits model size.
Bandwidth~120 GB/sTheoretical unified memory bandwidth.Excellent data throughput for shared resources.
Acceleration APIMetalApple's low-level graphics and compute framework.Used by LM Studio to interface with the M4 chip hardware.
Operating SystemmacOSNative Apple OS environment.Optimized for Apple Silicon architecture.

Network & Software Details​

  • Tailscale IP: 100.110.169.55
  • Primary Role: Efficient LLM Inference (Metal Backend).
  • Key Software: LM Studio v0.4.x (Utilizes Metal backend).

Operational Notes​

  • The unified memory architecture is a key advantage, allowing the OS and the model to share resources seamlessly without traditional PCIe bottlenecks.
  • Performance scaling is highly dependent on the specific optimization of the LLM framework for Apple's Metal API.

3. Control Node: vmi3537362 (Control VPS)​

This virtual private server acts as the central control plane, responsible for system orchestration, scheduling tasks, and managing persistent data ingestion from various sources. It does not perform direct LLM inference but manages the compute nodes.

Hardware Specifications​

ComponentSpecificationDetailsNotes
vCPUs6 vCPUsVirtualized CPU cores.Sufficient for running multiple background services concurrently.
RAM12 GB RAMDedicated system memory.Adequate for database caching and service overhead.
Operating SystemUbuntu 24.04 LTSLinux distribution, stable release.Standard environment for server-side tooling (Python, Node.js, etc.).

Network & Software Details​

  • Tailscale IP: (Assigned upon deployment)
  • Primary Role: Orchestration, Scheduling, Data Ingestion, and Gateway Management.
  • **Key Services