Deduplication
Improving Performance and Extending Tool Lifespan
Tapping multiple points across a network can result in the same packet being copied several times, particularly with East/West traffic and traffic from SPAN ports. This can negatively affect the performance of various tools as they not only need to process the same traffic multiple times, but in many cases further expend processing resources performing deduplication themselves. In instances where packet data is being recording to storage or a ring buffer, duplicate packets simply consume space without additional benefit. In some cases, the duplicate packets could even cause false positives or skew reporting.
Advanced Deduplication with Advanced Network Packet Broker
Cubro provides advanced, hardware-driven deduplication options tailored to user-selectable parameters, ensuring duplicate packets are eliminated before traffic is forwarded to your monitoring and security tools. This gives enterprise customers the flexibility to scale port density and bandwidth without over-provisioning budgets.
By offloading this resource-intensive task at the packet broker level, you reclaim critical processing power on downstream tools, allowing them to operate at peak efficiency.
High-Performance Packet Deduplication
Our advanced Network Packet Brokers (NPBs) deliver line-rate deduplication across high-speed 1/10/25/40/100Gbps links. This powerful feature eliminates redundant data streams at the ingress stage, protecting analytics, forensics, and monitoring equipment from severe traffic overloads.
How the Deduplication Engine Works
- Full-Packet Inspection: When the deduplication function is enabled, the packet broker calculates a highly accurate cryptographic checksum (hash-key) utilizing the entire network packet.
- Zero False Positives: Evaluating every single bit ensures the resulting hash represents the absolute maximum uniqueness of each packet. This prevents distinct packets that vary by only a single payload bit or deep header field from being erroneously discarded – a common issue with algorithms that only inspect a subset of the data.
- Line-Rate MD5 Hashing: The underlying hardware architecture generates a precise 16-byte MD5 hash-key for every single packet passing through the system.
- Stateful Memory Verification: These hash-keys are stored dynamically in high-speed onboard memory. Every incoming packet is checked against the active memory cache in real time.
- Intelligent Drop Logic: If an arriving packet’s hash matches an existing record within your user-defined time window, the duplicate packet is instantly dropped. Only unique traffic is permitted to traverse the output ports.
The Result: Your monitoring fabric receives clean, deduplicated, and highly optimized traffic streams – maximizing tool ROI, lowering data ingestion costs, and preventing false-positive analysis errors.
Benefits of Cubro Deduplication Solution
- Improved efficiency of monitoring and analytic tools.
- Maintain accuracy of tool reporting
- Offload function from tools to conserve processing resources
Data Deduplication
See how Cubro eliminates redundancy at the Ingress
Deduplication must be enabled on the Ingress inside the CPU.
Within the GHC Function, go inside a CPU Policy and navigate to the Ingress by clicking on the desired group. Inside, you can enable Deduplication in the Advanced Features. Once your selection is confirmed, the deduplication runs automatically; no further configuration is required to decapsulate your traffic.
Additional Resources
Our newsletter provides thought leadership content about the industry. It is concise and has interesting content to keep you updated with what’s new at Cubro and in the industry. You can unsubscribe anytime with a single click.
Your e-mail address is only used to send you our newsletter and information about the activities of Cubro Network Visibility. You can always use the unsubscribe link included in the newsletter.