“When your core link hits 95% utilization, customers don’t open tickets—they call your CEO. You need to see the bandwidth spike before it happens.”

In early 2013 at Net4 India, we managed over 1,000 active switch ports and router interfaces across two data centers.

Before modern metrics engines like Datadog, Prometheus, or Grafana existed, we relied on SNMP (Simple Network Management Protocol) and Cacti (RRDtool) for network telemetry.

The Poller Bottleneck

As we added hundreds of new customer switch ports, our Cacti poller started failing.

The standard PHP cmd.php poller took 7 minutes to complete a polling cycle. Because our polling interval was set to 5 minutes, poller processes stacked up, maxing out CPU and corrupting RRD (Round Robin Database) files.

[!IMPORTANT] When your telemetry polling cycle takes longer than your reporting interval, metrics flatline. You become blind to traffic micro-bursts.

# # System log showing poller overlap in Cacti
# tail -n 20 /var/log/cacti/cacti.log
  POLLER: Poller[0] MAXIMUM RUNTIME EXCEEDED! OVERLAPPING POLLER PROCESS DETECTED!
  SYSTEM STATS: Time:412.3451 s, Method:cmd.php, Processes:1, Threads:N/A, Hosts:240

The Optimization: Spine Poller & SNMPv2c Bulk Walks

We executed a complete telemetry poller overhaul:

1. Migrated to Spine: Replaced the single-threaded PHP cmd.php poller with Spine, a high-performance C-based multi-threaded polling daemon.
2. Upgraded to SNMPv2c: Replaced legacy SNMPv1 queries with SNMPv2c Bulk Requests, fetching 20 OIDs per packet.
3. Optimized RRD Storage: Placed RRD database files on a tmpfs RAM disk to eliminate disk I/O bottlenecks.

# # Spine Poller Configuration (/etc/spine.conf)
DB_Host       localhost
DB_Database   cacti
DB_User       cactiuser
DB_Pass       cactipass
DB_Port       3306

# Threads per process tuned for 8-core CPU
Threads       15

The Results

• Polling Duration: Reduced total polling cycle time from 412 seconds down to 28 seconds.
• Metric Accuracy: Captured 5-minute traffic spikes and interface error increments in real time.

The Verdict

Key Takeaway

Telemetry Pipelines Must Outperform Data Generation Rates.

Whether using SNMP/Cacti in 2013 or OpenTelemetry/Prometheus today, your observability poller must execute significantly faster than your collection window to prevent metric loss.