-
Notifications
You must be signed in to change notification settings - Fork 66
Initial pass at ingress-bgp docs #647
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| --- | ||
| title: Ingress Mesh BGP | ||
| date: 2025-01-26 | ||
| tier: sandbox | ||
| summary: This pattern demonstrates multi-cluster service mesh networking using MetalLB with BGP and Red Hat Service Interconnect (Skupper) for cross-cluster connectivity. | ||
| rh_products: | ||
| - Red Hat OpenShift Container Platform | ||
| - Red Hat Advanced Cluster Management | ||
| - Red Hat Service Interconnect | ||
| industries: | ||
| - General | ||
| - Telecommunications | ||
| aliases: /ingress-mesh-bgp/ | ||
| # pattern_logo: ingress-mesh-bgp.png # TODO: Create pattern logo | ||
| links: | ||
| github: https://github.com/validatedpatterns/ingress-mesh-bgp | ||
| install: imbgp-getting-started | ||
| bugs: https://github.com/validatedpatterns/ingress-mesh-bgp/issues | ||
| ci: ingressmeshbgp | ||
| --- | ||
| :toc: | ||
| :imagesdir: /images | ||
| :_content-type: ASSEMBLY | ||
| include::modules/comm-attributes.adoc[] | ||
|
|
||
| include::modules/imbgp-about.adoc[leveloffset=+1] | ||
|
|
||
| include::modules/imbgp-architecture.adoc[leveloffset=+1] | ||
|
|
||
| [id="next-steps_imbgp-index"] | ||
| == Next steps | ||
|
|
||
| * link:imbgp-getting-started[Deploy the Ingress Mesh BGP pattern]. |
21 changes: 21 additions & 0 deletions
21
content/patterns/ingress-mesh-bgp/imbgp-getting-started.adoc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| --- | ||
| title: Getting started | ||
| weight: 10 | ||
| aliases: /ingress-mesh-bgp/imbgp-getting-started/ | ||
| --- | ||
|
|
||
| :toc: | ||
| :imagesdir: /images | ||
| :_content-type: ASSEMBLY | ||
| include::modules/comm-attributes.adoc[] | ||
|
|
||
| include::modules/imbgp-deploying.adoc[leveloffset=1] | ||
|
|
||
| [id="next-steps_imbgp-getting-started"] | ||
| == Next steps | ||
|
|
||
| After the pattern is deployed and working correctly, you can: | ||
|
|
||
| * Verify the BGP routing is functioning correctly by checking the routing table on the core router | ||
| * Test the anycast service by accessing it from the client VM | ||
| * Explore the Red Hat Service Interconnect (Skupper) console to see the cross-cluster connectivity |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| :_content-type: CONCEPT | ||
| :imagesdir: ../../images | ||
|
|
||
| [id="about-ingress-mesh-bgp-pattern"] | ||
| = About the Ingress Mesh BGP pattern | ||
|
|
||
| Use case:: | ||
|
|
||
| * Deploy multi-cluster applications with unified ingress using BGP-based load balancing. | ||
| * Enable anycast IP addressing for seamless failover between OpenShift clusters. | ||
| * Connect services across clusters using Red Hat Service Interconnect (Skupper) for secure east-west traffic. | ||
| * Leverage Kubernetes Gateway API to give application developers self-service control over service routing without depending on network infrastructure teams. | ||
| * Demonstrate enterprise-grade BGP routing integration with Kubernetes/OpenShift environments. | ||
| + | ||
| [NOTE] | ||
| ==== | ||
| This pattern is designed for AWS environments and simulates a datacenter-like BGP network topology using EC2 instances as routing infrastructure. | ||
| ==== | ||
|
|
||
| Background:: | ||
| Modern distributed applications often span multiple clusters for high availability, geographic distribution, or workload isolation. Traditional ingress solutions provide per-cluster access, but organizations need unified entry points that can intelligently route traffic across clusters. | ||
|
|
||
| This pattern addresses these requirements by combining: | ||
|
|
||
| * *MetalLB with BGP mode* - Provides load balancer services that advertise routes via BGP, enabling anycast addressing where the same IP is reachable through multiple clusters. | ||
mbaldessari marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| * *Gateway API* - Delivers L4 and L7 service routing via next generation Ingress, Load Balancing, and Service Mesh APIs. Provides GatewayClasses (infrastructure) and Gateways (operations) so application developers can create routes (HTTPRoute, GRPCRoute, etc.) for their services. | ||
| * *Red Hat Service Interconnect (Skupper)* - Creates a secure application-layer virtual network connecting services across clusters without requiring VPN or special network configurations. | ||
| * *FRRouting (FRR)* - Industry-standard routing software running on EC2 instances that acts as the BGP peering infrastructure, simulating top-of-rack (TOR) switches and core routers. | ||
|
|
||
| Gateway API plays a central role in this architecture. It is the intermediary layer between BGP and Skupper: traffic arriving at a cluster via BGP-advertised anycast IPs is routed through Gateway API down to the appropriate Skupper site, where Skupper handles inter-cluster routing for sparse deployments or services not locally available. Gateway API also separates concerns across organizational roles — infrastructure teams define GatewayClasses, operations teams create Gateways, and application developers independently manage their own Routes (HTTPRoute, GRPCRoute, etc.) for both simple routing and more advanced mesh-like routing scenarios. | ||
|
|
||
| [id="about-solution-imbgp"] | ||
| == About the solution | ||
|
|
||
| This pattern deploys a complete multi-cluster networking demonstration on AWS that includes: | ||
|
|
||
| * Two OpenShift clusters, designated as "west" (ACM hub) and "east" (spoke) | ||
| * A simulated routing infrastructure with FRR-based routers | ||
| * MetalLB configured in BGP mode on both clusters | ||
| * Red Hat Service Interconnect linking services between clusters | ||
| * A hello-world application deployed across both clusters demonstrating the connectivity | ||
|
|
||
| The solution uses ECMP (Equal-Cost Multi-Path) routing to distribute traffic across both clusters when accessing the anycast IP address. | ||
|
|
||
| .Benefits of the Ingress Mesh BGP pattern: | ||
|
|
||
| * *Unified service access* - Single IP address reaches services on multiple clusters | ||
| * *Automatic failover* - BGP route withdrawal provides fast failover when a cluster becomes unavailable | ||
| * *Secure cross-cluster communication* - Skupper encrypts all inter-cluster traffic using mutual TLS | ||
| * *No network infrastructure changes* - Skupper works over existing networks without VPNs or firewall changes | ||
| * *GitOps-driven deployment* - All components are deployed and managed through ArgoCD | ||
| * *Application owner autonomy* - App owners can describe their own routes on approved gateways and gatewayclasses without relying on network infrastructure teams | ||
|
|
||
mbaldessari marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| [id="about-technology-imbgp"] | ||
| == About the technology | ||
|
|
||
| The following technologies are used in this solution: | ||
|
|
||
| https://www.redhat.com/en/technologies/cloud-computing/openshift/try-it[Red Hat OpenShift Platform]:: | ||
| An enterprise-ready Kubernetes container platform built for an open hybrid cloud strategy. It provides a consistent application platform to manage hybrid cloud, public cloud, and edge deployments. | ||
|
|
||
| https://www.redhat.com/en/technologies/management/advanced-cluster-management[Red Hat Advanced Cluster Management for Kubernetes]:: | ||
| Controls clusters and applications from a single console, with built-in security policies. Extends the value of Red Hat OpenShift by deploying apps, managing multiple clusters, and enforcing policies across multiple clusters at scale. | ||
|
|
||
| https://www.redhat.com/en/technologies/cloud-computing/service-interconnect[Red Hat Service Interconnect]:: | ||
| Based on the open source Skupper project, Red Hat Service Interconnect enables secure communication between services across different environments without requiring VPN infrastructure or special firewall rules. It creates a virtual application network that works at Layer 7. | ||
|
|
||
| https://metallb.universe.tf/[MetalLB]:: | ||
| A load-balancer implementation for bare metal Kubernetes clusters, using standard routing protocols. In this pattern, MetalLB operates in BGP mode to advertise service IPs as routes to the upstream network infrastructure. | ||
|
|
||
| https://frrouting.org/[FRRouting (FRR)]:: | ||
| A free and open source Internet routing protocol suite for Linux and Unix platforms. It implements BGP, OSPF, RIP, and other protocols. In this pattern, FRR runs on EC2 instances to simulate datacenter routing infrastructure. | ||
|
|
||
| https://gateway-api.sigs.k8s.io/[Kubernetes Gateway API]:: | ||
| The next generation of Kubernetes Ingress, providing a more expressive and extensible API for managing traffic into and within a cluster. Gateway API is the intermediary layer between BGP and Skupper — traffic arriving via BGP passes through Gateway API for routing before reaching Skupper for inter-cluster communication. It provides GatewayClasses for infrastructure providers, Gateways for operations teams, and Routes (HTTPRoute, GRPCRoute, etc.) for application developers, enabling self-service routing without relying on network infrastructure teams. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,152 @@ | ||
| :_content-type: CONCEPT | ||
| :imagesdir: ../../images | ||
|
|
||
| [id="architecture-ingress-mesh-bgp"] | ||
| = Architecture | ||
|
|
||
| The Ingress Mesh BGP pattern demonstrates a multi-cluster networking architecture that combines BGP-based anycast ingress with service mesh connectivity. The architecture simulates an enterprise network topology on AWS. | ||
|
|
||
| [id="network-topology-imbgp"] | ||
| == Network topology | ||
|
|
||
| The pattern creates the following network components on AWS: | ||
|
|
||
| .Network topology diagram | ||
| image::ingress-mesh-bgp/network.svg[Network Topology,700] | ||
|
|
||
| === VPCs and subnets | ||
|
|
||
| The pattern provisions several VPCs to simulate separate network segments: | ||
|
|
||
| * *Client-Core VPC (192.168.8.0/24)* - Contains the client VM and core router | ||
| * *Core-West TOR VPC (192.168.12.0/24)* - Connects the core router to the west cluster's top-of-rack router | ||
| * *Core-East TOR VPC (192.168.16.0/24)* - Connects the core router to the east cluster's top-of-rack router | ||
| * *West Workers VPC (10.0.0.0/16)* - The west OpenShift cluster's VPC | ||
| * *East Workers VPC (10.1.0.0/16)* - The east OpenShift cluster's VPC | ||
|
|
||
| === Routing infrastructure | ||
|
|
||
| The pattern deploys EC2 instances running FRRouting to create a simulated datacenter network: | ||
|
|
||
| [cols="1,1,2"] | ||
| |=== | ||
| |Component |ASN |Description | ||
|
|
||
| |Core Router | ||
| |64666 | ||
| |Central router that peers with both TOR routers and advertises client network routes | ||
|
|
||
| |West TOR | ||
| |64001 | ||
| |Top-of-rack router for the west cluster, peers with core and west OpenShift workers | ||
|
|
||
| |East TOR | ||
| |64002 | ||
| |Top-of-rack router for the east cluster, peers with core and east OpenShift workers | ||
|
|
||
| |West OpenShift (MetalLB) | ||
| |65001 | ||
| |MetalLB speakers on west cluster workers, peer with west TOR | ||
|
|
||
| |East OpenShift (MetalLB) | ||
| |65002 | ||
| |MetalLB speakers on east cluster workers, peer with east TOR | ||
| |=== | ||
|
|
||
| === Anycast addressing | ||
|
|
||
| Both clusters advertise the same anycast IP range (192.168.155.0/24) via BGP. When a client accesses an anycast IP: | ||
|
|
||
| .Client path to services via anycast and BGP | ||
| image::ingress-mesh-bgp/ingress-path.svg[Client Ingress Path,700] | ||
|
|
||
| . The core router receives BGP advertisements from both TOR routers for the anycast range | ||
| . ECMP routing distributes traffic across both paths | ||
| . Requests reach either the west or east cluster based on the routing decision | ||
| . If one cluster becomes unavailable, BGP route withdrawal automatically redirects traffic to the remaining cluster | ||
|
|
||
| [id="cluster-components-imbgp"] | ||
| == Cluster components | ||
|
|
||
| === West cluster (Hub) | ||
|
|
||
| The west cluster acts as the management hub and includes: | ||
|
|
||
| * *Red Hat Advanced Cluster Management* - Manages the east cluster as a spoke | ||
| * *HashiCorp Vault* - Centralized secrets management | ||
| * *External Secrets Operator* - Synchronizes secrets from Vault to Kubernetes | ||
| * *MetalLB* - Provides BGP-advertised load balancer services (ASN 65001) | ||
| * *Gateway API* - Routes incoming traffic to appropriate services, providing the intermediary layer between BGP ingress and Skupper | ||
| * *Red Hat Service Interconnect (Skupper)* - Hosts the Skupper site with link access enabled | ||
mbaldessari marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| * *Hello-world application* - Frontend component of the demo application | ||
|
|
||
| === East cluster (Spoke) | ||
|
|
||
| The east cluster is a managed spoke that includes: | ||
|
|
||
| * *External Secrets Operator* - Retrieves secrets from the hub's Vault | ||
| * *MetalLB* - Provides BGP-advertised load balancer services (ASN 65002) | ||
mbaldessari marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| * *Gateway API* - Routes incoming traffic to appropriate services, providing the intermediary layer between BGP ingress and Skupper | ||
| * *Red Hat Service Interconnect (Skupper)* - Connects back to the west cluster's Skupper site | ||
| * *Hello-world application* - Backend component of the demo application | ||
|
|
||
| [id="metallb-imbgp"] | ||
| == MetalLB | ||
|
|
||
| MetalLB provides load balancer services on bare metal and cloud environments where cloud-native load balancers are not available or not suitable: | ||
|
|
||
| * Each cluster runs MetalLB in BGP mode with a unique ASN (65001 for west, 65002 for east) | ||
| * MetalLB speakers on worker nodes peer with the local TOR router and advertise service IPs via BGP | ||
| * Both clusters advertise the same anycast IP range (192.168.155.0/24), enabling ECMP routing from the core | ||
| * When a cluster becomes unavailable, its BGP routes are withdrawn and traffic is automatically redirected to the remaining cluster | ||
|
|
||
| [id="gateway-api-imbgp"] | ||
| == Gateway API | ||
|
|
||
| Gateway API provides the L4/L7 routing layer between BGP ingress and application services: | ||
|
|
||
| * *GatewayClass* - Defined by infrastructure providers to describe the type of gateway infrastructure available | ||
| * *Gateway* - Created by operations teams to instantiate a gateway from a GatewayClass, defining listeners and allowed routes | ||
| * *HTTPRoute / GRPCRoute* - Created by application developers to describe how traffic should be routed to their services | ||
|
|
||
| Gateway API is the intermediary step between BGP and Skupper. Traffic arriving at a cluster via BGP-advertised anycast IPs passes through Gateway API for service routing before reaching Skupper for inter-cluster communication. This separation of concerns allows application developers to define their own routing rules on approved gateways without relying on network infrastructure teams. | ||
|
|
||
| [id="service-interconnect-imbgp"] | ||
| == Red Hat Service Interconnect | ||
mbaldessari marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| Red Hat Service Interconnect (based on Skupper) creates a virtual application network between the clusters: | ||
|
|
||
| * The west cluster hosts a Skupper site with `linkAccess: default`, allowing other sites to connect | ||
| * The east cluster establishes a link to the west cluster using a pre-shared access token | ||
| * Services exposed through Skupper listeners become accessible across both clusters | ||
| * All traffic between sites is encrypted using mutual TLS | ||
|
|
||
| The pattern uses the Skupper v2 API with the following components: | ||
|
|
||
| * *Site* - Defines the Skupper installation in each namespace | ||
| * *Listener* - Exposes a service to the Skupper network | ||
| * *Connector* - Connects a local workload to a Skupper-exposed service | ||
| * *AccessGrant/AccessToken* - Manages secure connection between sites | ||
|
|
||
| [id="gitops-structure-imbgp"] | ||
| == GitOps structure | ||
|
|
||
| The pattern follows the Validated Patterns framework: | ||
|
|
||
| ---- | ||
| ingress-mesh-bgp/ | ||
| ├── values-global.yaml # Global configuration | ||
| ├── values-west.yaml # West (hub) cluster configuration | ||
| ├── values-east.yaml # East (spoke) cluster configuration | ||
| ├── charts/ | ||
| │ ├── all/ | ||
| │ │ ├── hello-world/ # Demo application | ||
| │ │ ├── metallb/ # MetalLB configuration | ||
| │ │ └── rhsi/ # Skupper configuration for west | ||
| │ └── east-site/ | ||
| │ └── rhsi-east/ # Skupper configuration for east | ||
| └── ansible/ | ||
| └── playbooks/ # Infrastructure automation | ||
| ---- | ||
|
|
||
| ArgoCD manages the deployment of all components, with Red Hat Advanced Cluster Management distributing configurations to the appropriate clusters. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.