A practical guide to direct site connectivity, scalable topology choices, traffic behavior, and Ray implementation concepts

1. Brief overview

The live Ray configuration article demonstrates how two SD-WAN members can join a mesh network, advertise their local LAN subnets, and forward traffic directly between locations. This rewritten guide removes the portal screenshots and focuses on the architecture: what mesh means, how packets travel, where it helps, and when full mesh, partial mesh, or relay is the better design.

2. What is an SD-WAN mesh?

An SD-WAN mesh is an overlay topology in which sites are allowed to establish direct encrypted tunnels with other mesh members. Each site still uses an underlay such as broadband, dedicated internet, MPLS, LTE/5G, or a cloud backbone. The overlay uses those transports to carry protected enterprise routes and application traffic.

• Underlay: the physical or provider IP transport that makes tunnel endpoints reachable.

• Overlay: the encrypted logical fabric that carries branch, data-center, and cloud traffic.

• Control and policy: the information that determines which sites may peer, which routes are visible, and which path an application should use.

• Data plane: the actual user packets flowing directly between sites or through a relay.

3. Full mesh, partial mesh, and relay

Figure 1. Three ways to organize connectivity between SD-WAN sites.

1. Full mesh

Every site can form a direct tunnel to every other site. This gives the shortest overlay path for branch-to-branch traffic and works well when many sites exchange voice, video, files, database traffic, or other latency-sensitive data. The trade-off is scale: for n sites, a theoretical full mesh contains n(n-1)/2 site pairs before counting multiple WAN transports per pair.

2. Partial mesh

Only selected site pairs form direct tunnels. Important regional offices, warehouses, call centers, data centers, or collaboration-heavy branches can communicate directly, while other traffic follows a hub or relay. Partial mesh balances performance with tunnel, route, session, security-policy, and operational scale.

3. Relay

A relay is an intermediate SD-WAN node that forwards traffic between sites that do not have, cannot establish, or are not permitted to use a direct tunnel. The relay might be a hub, regional gateway, transport gateway, or another designated transit node. Relay behavior, automatic fallback, and dynamic shortcut support are product- and release-dependent and must be validated for Ray before publication as a guaranteed feature.

4. How direct and relayed traffic differ

Figure 2. A direct mesh tunnel minimizes hops; a relay provides an indirect transit path when designed and supported.

5. Where mesh helps

• Branches frequently exchange files or access distributed storage.

• Voice and video calls run directly between offices.

• Factories, warehouses, and stores communicate with regional applications or each other.

• Data replication occurs between data centers or cloud locations.

• A centralized hub would create avoidable latency, bandwidth cost, or congestion.

• Sites need alternate direct paths across multiple WAN providers.

6. Problems mesh can solve

7. Choosing the right topology

8. Ray implementation overview

The Ray workflow begins at the parent cluster so appliances in child clusters can be selected as members. The detailed portal screenshots are intentionally omitted here; the essential configuration decisions are:

• Create an SD-WAN network and select MESH topology.

• Choose a unique tunnel CIDR so each member receives an overlay address.

• Select the default SLA, MTU, and cipher such as AES or ChaChaPoly according to the deployment standard.

• Add each appliance as a mesh member and supply a public endpoint only where required by reachability and NAT design.

• Select the WAN links and optimization algorithm appropriate for the application; packet cloning should be reserved for traffic whose reliability benefit justifies duplicate bandwidth.

• Advertise only the non-overlapping local LAN subnets that other members must reach.

• Set outbound forwarding toward the LAN for traffic entering a member site, or toward the WAN only for an explicitly designed breakout use case.

• Create and validate the necessary SLA and SD-WAN traffic rules, then test both directions.

9. Routing and reachability requirements

A tunnel being up does not guarantee that applications can communicate. Each source site needs a selected route to the destination LAN prefix, the destination site needs a return route, and firewall or segmentation policy must allow the flow.

• Keep every advertised LAN subnet unique, or design deliberate translation for overlapping addresses.

• Ensure the underlay can reach peer or relay tunnel endpoints through upstream firewalls and NAT.

• Use specific route advertisement rather than exposing all networks by default.

• Confirm that more-specific routes do not send traffic around the intended overlay path.

• For relayed traffic, the relay must learn both sides and permit transit forwarding.

10. SLA and path selection

With multiple WAN circuits, a site pair may have several candidate overlay tunnels. SLA probes measure reachability, latency, jitter, and packet loss. Traffic policy can then choose a best path, distribute sessions by weight, keep a secondary path on standby, clone selected packets, or—where appropriate—distribute packets across links.

11. Security considerations

Full mesh improves reachability but can enlarge the security blast radius if every site learns every route. Encryption protects traffic in transit, but it does not replace segmentation, firewall policy, endpoint security, identity controls, or logging.

• Permit only the site pairs and prefixes required by the business.

• Separate users, servers, guests, IoT, voice, and management into appropriate segments.

• Decide where inspection occurs when traffic bypasses the central hub.

• Protect management and control-plane access with strong authentication and least privilege.

• Track direct and relayed flows so investigations can identify the real path.

12. Scale, MTU, and performance

As sites and WAN links increase, the number of potential tunnels, routes, SLA sessions, and encryption/security associations increases. Confirm appliance and controller limits before choosing full mesh. Encapsulation also reduces usable MTU; mismatches can produce working pings but failed transfers or stalled TLS sessions.

• Use partial mesh or regional groups when full-mesh scale is unnecessary.

• Account for every WAN transport when estimating tunnel counts.

• Size relay and hub nodes for aggregate transit throughput and session load.

• Set overlay MTU for tunnel overhead and test path MTU across broadband, MPLS, LTE/5G, and cloud circuits.

• Allow required path-MTU discovery behavior where security policy permits.

13. Failure scenarios

14. Validation checklist

• All intended members have unique tunnel addresses and healthy status.

• Direct peers can reach each other over every intended underlay.

• Advertised LAN prefixes are unique, selected, and visible only where required.

• Bidirectional host tests succeed for direct and relayed traffic paths.

• Trace route confirms whether the session is direct or passes through a relay/hub.

• SLA measurements match observed latency, loss, and jitter.

• Failover and failback work without unstable path flapping.

• Packet cloning or per-packet behavior is enabled only for justified applications.

• Mesh appliances, controllers, and any relays remain within supported tunnel, route, and throughput scale.

• Logs and monitoring identify the member, underlay, tunnel, and policy used by each test.

15. When mesh is not the best choice

Use hub-and-spoke or central service insertion instead when most traffic is branch-to-data-center, every inter-site flow must cross a central firewall, branch appliances cannot support the required peer scale, or the organization prioritizes strict centralized control over the shortest path. A partial mesh is often the practical middle ground.

16. Conclusion

Mesh SD-WAN improves site-to-site performance by creating direct overlay paths over diverse underlays. Full mesh offers maximum direct reachability, partial mesh limits direct connectivity to useful site groups, and relay provides an indirect transit concept when direct communication is unavailable or intentionally controlled. The best design aligns topology with applications, security, device scale, NAT reachability, and failure requirements rather than enabling every possible tunnel by default.

17. Extended Topology Design Guide

The following deep dives explain how each topology behaves in production, where it is useful, and what an operator must accept in return. A mesh decision should be based on application flows, security boundaries, appliance scale, underlay reachability, and failure behavior—not simply on the availability of a Mesh option in the controller.

18. Full mesh: detailed behavior

In a full mesh, every participating site is permitted to communicate directly with every other participating site. The source edge learns the destination LAN route and the available transport paths to that destination. It applies segmentation, security, and SLA policy, encapsulates the packet, and sends it over a selected direct tunnel. The receiving edge decrypts the packet and forwards it to its LAN. A central hub does not carry the ordinary site-to-site payload.

Direct does not necessarily mean one physical path. If two sites each have broadband and MPLS, several overlay path combinations may exist. The SD-WAN policy can prefer a carrier, use the path with the best latency or loss, distribute sessions by weight, or keep one path on standby. Consequently, the real amount of path state can be much larger than the number of site pairs.

1. Full-mesh use cases

• Voice and video between branches, where every unnecessary transit hop adds delay and jitter.

• Large file exchange, media workflows, backups, and distributed storage that would otherwise consume hub bandwidth twice.

• Data-center or cloud-edge replication requiring direct high-capacity paths and fast failover.

• Manufacturing, logistics, or retail locations that consume services hosted at several peer sites.

• Distributed application clusters with significant east-west traffic between locations.

2. Full-mesh advantages

• Shortest overlay path between sites and usually the lowest avoidable latency.

• No central transit bandwidth is consumed for normal peer traffic.

• Lower dependency on one hub for site-to-site reachability.

• Multiple underlays can be evaluated separately for every site pair.

• Works naturally when users and applications are widely distributed.

3. Full-mesh disadvantages

• Tunnel relationships grow quickly: 10 sites produce 45 site pairs, 50 produce 1,225, and 100 produce 4,950 before multiple transports are counted.

• Edges and controllers carry more routes, peers, security associations, SLA sessions, and telemetry.

• Hub-only firewalls no longer inspect direct traffic; equivalent distributed or cloud security may be required.

• Troubleshooting becomes path-specific because applications may select different peers and underlays.

• Any-to-any route visibility can increase the security blast radius unless segmentation is strict.
• Restrictive NAT and upstream firewalls may prevent some direct peer relationships.

19. Partial mesh: detailed behavior

Partial mesh creates direct tunnels only between selected sites or groups. The selection can follow geography, business function, traffic volume, application dependency, compliance boundary, or edge-device capacity. Traffic outside the selected relationships uses a hub, regional gateway, relay, or another approved path.

1. Ways to build a partial mesh

• Regional mesh: sites inside a country or region communicate directly, while regional gateways connect the groups.

• Application community: factories mesh with engineering sites; stores mesh with warehouses; other sites remain hub-and-spoke.

• Core-site mesh: data centers, cloud edges, and large campuses form a mesh while small branches connect to one or two core sites.

• Selective shortcuts: only branch pairs with high collaboration demand receive direct connectivity.

• Segment-specific topology: voice uses mesh while regulated payment traffic remains on a security-hub path.

2. Partial-mesh advantages

• Captures the latency and bandwidth benefits where they matter without building every possible tunnel.

• Keeps sensitive traffic on centrally inspected paths.

• Protects smaller branch appliances from unnecessary peer state.

• Supports regional and data-residency boundaries.

• Aligns connectivity with measured communication patterns.

3. Partial-mesh disadvantages

• Requires accurate application and traffic-flow knowledge.

• Topology and route policy are more complex than a uniform design.

• Traffic patterns change, so site groups can become stale.

• Operators must know which flows are direct and which are relayed.
• Site moves and new applications require coordinated topology, segmentation, and route updates.

20. Relay: detailed behavior

A relay is an intermediate data-plane node. It forwards traffic when sites do not have, cannot establish, or are not allowed to use a direct path. A relay can be a hub, regional gateway, transport gateway, or another designated transit node. The source sends the packet through one overlay leg to the relay; the relay performs a route and policy decision and sends it through a second leg to the destination.

1. Why relay may be required

• NAT traversal: a site cannot accept a direct inbound tunnel because of upstream NAT or firewall behavior.

• Transport mismatch: two private networks cannot reach each other, but the relay connects to both.

• Security insertion: a firewall, IDS/IPS, proxy, or compliance service must inspect the flow.

• Regional interconnection: two partial-mesh regions exchange traffic through their gateways.

• Scale control: small branches avoid maintaining direct tunnels to every destination.

• Fallback: an unhealthy direct path is replaced by an indirect transit path, where supported.

2. Relay advantages

• Provides connectivity when direct peering is unavailable.

• Creates a convenient enforcement point for security services.

• Reduces peer scale on small edges.

• Connects different regions or underlay environments.

• Can provide a backup path for selected failures.

3. Relay disadvantages

• Adds latency and jitter because packets take an extra overlay hop.

• Consumes relay WAN bandwidth and processing on both tunnel legs.

• Creates a shared failure and congestion domain unless redundant relays exist.

• May break stateful services if forward and return traffic use different relays.

• Requires additional path visibility and troubleshooting skills.

• Automatic fallback and dynamic shortcut behavior are product- and release-dependent and require Ray validation.