Julius Kaiser
Leipzig, Germany
jkdata@mailbox.org


Title: How to become...
Status: Release

Published: 27 Oct 2023
Updated: 27 Oct 2023

How to become an Internet service provider (ISP)

Getting public IP address ranges routed on the Internet is no rocket science. It does, however, involve steps (including paperwork!) and concepts that are probably unfamiliar to administrators who are used to private routing domains only. This guide tries to explain what’s essential to register public IP prefixes for an organisation and make them reachable globally.

The Internet 2003 from opte.org ("The Internet 2003" | Source: opte.org)

Table of Contents

Intention & Intended Audience

I hope this text can shed a few lights on a topic that is otherwise invisible to most people - even to most network administrators. If you are working with IP networks and plan to multihome your connection to the Internet, this guide hopefully helps your understanding on how regulatory (in the sense of global network coordination) and technical aspects are working together.

Even if you do not have a technical background, this text may draw a picture of what is needed to create the Internet - the global network of networks that connects about five billion human beings and more than 25 billion devices, with a fast growing tendancy.1

This is not an in-depth guide on any of the topics involved. It’s about connecting the dots and being detailed enough to give clear guidance towards establishing a new network on the Internet.

The terms “you” and “your organisation” are pretty much used interchangeably in this text.

Status of This Document

All essential steps to establish a new network and announce it on the Internet are contained with the initial release of this guide. I’m now just trying to release early and release often and would appreciate any constructive feedback.

Here’s a list of aspects that shall be elaborated further over the next few weeks and months:

  • Public Peering at IXPs
  • Selection criteria for transit providers
  • Selection criteria for network equipment
  • BGP functionality and operation
  • Example router and BGP configurations
  • BGP and routing with open software
  • Route origin validation (IRRDB filtering and RPKI)
  • How to become a transit ISP
  • Analysing BGP operation in the DFZ
  • Musings on the last mile

ISP - Definitions & Classifications

What comes to most peoples minds when they hear “ISP” probably is an Internet access provider; An organisation that provides Internet access to private households and businesses, usually over technologies like digital subscriber line (DSL), cable or cellular networks.

In a broader sense, the term Internet service provider includes organisations that offer services like web hosting, e-mail, content streaming, domain name registration etc. pp.

All these use-cases rely on public IP addressing. If you’re taking care of that independently of third parties (by operating your own IP address space), this makes your organisation an ISP in the narrower sense of the Internet Numbers Registry System (RFC7020). This guide mainly deals with acquiring and fulfilling this role.

There’s also a commonly, yet not authoritatively defined classification of ISPs into tier-one to -three network providers.2

The Benefits of Being Your Own ISP

If you’re thinking about getting own IP address ranges, you either have niche interests or a certain amount of infrastructure that needs reliable connection to the Internet.

While highly available Internet access is a standard especially in datacenter environments and can just be obtained there in exchange for money, being your own ISP buys your organisation more independence from IP transit providers (other ISPs) and datacenter facilities:

Regardless of which ISPs are available in a certain facility, operating your own address space means that you only need them for carrying (transiting) your IP traffic to other networks - and vice versa - but not for serving you with IP addresses from the ranges that are assigned to them. Thus you are free to change transit providers and geographical locations without the need for readdressing your services or relying on points of presences (POPs) of your current ISPs. You just operate your own IP addresses wherever you need them.

Overview: Becoming Your Own ISP

To take part in global interdomain routing generally requires:

  • Requesting Internet number resources for your organisation
  • Maintaining Internet number registry data for your organisation
  • Maintaining Internet routing registry data for your resources
  • Physically connecting your network to IP transit providers
  • Announcing your IP prefixes on the Internet via BGP

The example data used in this guide is either taken from my personal AS47462 (which I operate for fun & educational purpose) or from IANAs special registries intended for documentation or local usage.3

Internet Number Resources and -Registries

Internet number resources include Internet Protocol (IP) address ranges (IPv4 & IPv6) and autonomous system numbers (ASNs). These are the essential building blocks for world-wide communication over the Internet Protocol. Both are governed by the Internet Assigned Numbers Authority (IANA), which delegates responsibility for larger blocks of these numbers to the five regional Internet registries (RIRs) - AFRINIC, APNIC, ARIN, LACNIC, and the RIPE NCC - each responsible for a different geographic region:

Regional Internet Registries world map
(Regional Internet Registries world map, Source: Wikimedia Commons)

These RIRs in turn sub-delegate seperate number resources to local Internet registries (LIRs): telcos, other enterprises, governments, NGOs - and possibly your organisation.

To register Internet number resources, an organisation either has to become a LIR itself or get resources sponsored from an existing LIR.

Both cases are covered in this guide.

The informational RFC 7020 provides approved high-level documentation of the current Internet Numbers Registry System.

Autonomous System Numbers (ASNs)

An autonomous system (AS) is an independent network, commonly under control of a single administrative entity. This can be a single person or a large, transnational corporation. An AS usually operates multiple public IP address ranges that share a common, external routing policy. An autonomous systems number (ASN) uniquely identifies such a set of networks and is used to share routing information between adjacent autonomous systems via the Border Gateway Protocol (BGP).

BGP is covered later in this guide.

The entirety of autonomous systems that are able to reach each other (at least for most of the time..) form what we call the Internet.

Internet Protocol (IP) Address Ranges

The information on address assignment and allocation policies summarised here refer to the situation in RIPE NCC governed regions. Please note that I am not speaking for the RIPE NCC. If uncertain, always refer to their official publications.

IP address ranges are aggregates of continious address space - called prefixes. Loosely compareable to telephone area codes.

RIRs differentiate between Provider-aggregatable (PA) and Provider-independent (PI) Internet Protocol address space with regard to their respective address assignment policies.

This affects which types of organisations can be assigned which type of address space and how this address space can be used.

See these two documents for all the details:

Here’s what that generally means in the RIPE NCC area:

Provider-Aggregatable (PA) Address Space

Provider-aggregatable (PA) address space can only be assigned to a LIR. To become a LIR your organisation has to become a member of the RIPE NCC. This is generally possible for any legal entity or natural person that is able to pay the initial sign-up and annual membership fees. See Become a RIPE NCC Member (RIPE).

PA space is “all features unlocked”: A LIR is allowed to sub-delegate IP address assignments (and the legal responsibility for them) to third parties. PA space can thus be used to offer broadband services to subscribers, collocation services or for addressing larger organisations4, including governments.5

Provider-Independent (PI) Address Space

Provider-independent (PI) address space can be assigned to so called “End User organisations” (again, possibly any legal entity or natural person unless too big).

PI space cannot be sub-assigned and is not meant to be used by larger organisations with seperate legal sub-entities. It is intended to be used for an organisations own infrastructure. This, however, includes usage of such addresses for publicly available “Software as a service” (SaaS) like webmail- and other portals, connecting customer servers & appliances with seperate addresses and even operating access networks like public hotspots, as long as each connecting device receives only seperate IP addresses - not prefixes. 4

Internet Number- & Routing Registry Databases

Distinguishing Internet number registry databases and Internet routing registry databases (IRRDBs) can be a bit confusing. Both services are overlapping in the databases of at least RIPE and APNIC.

Internet number registry databases contain things like legal registration- and contact details for Internet number resources and their holders. Internet routing registries contain information about how Internet traffic should be routed in terms of BGP routing policies and configurations.

Some database objects (like the aut-num object) have fields that serve purposes in both kinds of registries.

Take my AS47462 aut-num object for an example:

aut-num:         AS47462
as-name:         JKDATA-DE
org:             ORG-JA512-RIPE
import:          from AS200297 accept ANY
export:          to AS200297 announce AS47462
import:          from AS21413 accept ANY
export:          to AS21413 announce AS47462
sponsoring-org:  ORG-ASG32-RIPE
admin-c:         JKA
tech-c:          JKA
status:          ASSIGNED
mnt-by:          RIPE-NCC-END-MNT
mnt-by:          jkdata-de-mnt
created:         2008-06-23T11:25:16Z
last-modified:   2023-09-19T21:15:16Z
source:          RIPE

It links the autonomous system (aut-num) AS47462 to my organisation (org) ORG-JA512-RIPE - a typical function of an Internet number registry. You get to know who’s legally responsible for the AS and how to reach them. It also publicly documents the routing policy for AS47462 in the form of import- and export-statements, expressing which ASNs are accepted from and announced by AS47462.

Altough this information is not technically neccesary for routing in terms of the Border Gateway Protocol, almost all ISPs use information from IRRDBs to maintain their routing configurations.

That practically means that other ISPs won't accept prefix announcements from your AS via BGP unless you have route-objects for your prefixes that are in turn properly linked to your aut-num object. This is how basic routing security is implemented on a global scale; IRRDBs are a source of truth for that purpose.

Besides the databases operated by RIRs, there are also IRRDBs that are operated by other organisations; the earliest and probably most well-known of them is the Routing Assets Database (RADb).

Creating Essential RIPE Database Objects

If your organisation wants to get Internet number resource assignments from RIPE (either directly or via a sponsoring LIR), it needs to set up a few essential objects in the RIPE database.

First of all, create a free account at www.ripe.net. Next, create a ROLE and MNTNER as well as an ORGANISATION object. These objects are mandatory for all further steps involving the RIPE database.

The easiest way to create and edit objects in the RIPE DB is via the web-based interface located under my.ripe.net.

See these documents for a comprehensive description and explanation of all the available object types in the RIPE database:

Below are the ROLE, MNTNER and ORGANISATION objects for my personal, non-profit AS47462 as an example. Look up objects from other autononmous systems to get an idea of how organisations are using them. There can be quite some curiosities found out there..

ROLE Object

Here’s my general ROLE object. As you can see, it mainly serves the purpose of providing contact information.

role:            jlsksr
address:         Julius Kaiser, Fsociety, Karl-Liebknecht-Str 8, 04107 Leipzig, Germany
e-mail:          jkdata@posteo.de
phone:           +49 341 97854941
nic-hdl:         JKA
mnt-by:          jkdata-de-mnt
created:         2023-03-09T14:22:56Z
last-modified:   2023-03-09T14:25:37Z
source:          RIPE

“A role object is similar to a person object. However, instead of describing a single person, it describes a role performed by one or more people. This might be a help desk, network monitoring centre, team of system administrators, etc.” (Source: RIPE, Description of the ROLE Object)

MNTNER Object

This is the MNTNER (pronounced “maintainer”) object for all things AS47462. It is used to control who’s authorised to create, edit or delete objects that are linked to it.

mntner:          jkdata-de-mnt
admin-c:         JKA
upd-to:          jkdata@posteo.de
auth:            SSO juliuskaiser@posteo.net
mnt-by:          jkdata-de-mnt
created:         2022-02-07T05:17:01Z
last-modified:   2023-03-09T14:34:05Z
source:          RIPE

“Objects in the RIPE Database are protected using mntner objects. A mntner object is an anonymous box containing the credentials needed to authorise creation, deletion or modification of any objects that it protects by whomever maintains this data.” (Source: RIPE, Description of the MNTNER Object)

ORGANISATION Object

And here’s my ORGANISATION object.

organisation:    ORG-JA512-RIPE
org-name:        Julius Kaiser
country:         DE
org-type:        OTHER
address:         Julius Kaiser
address:         c/o Fsociety
address:         Karl-Liebknecht-Str 8
address:         04107 Leipzig
e-mail:          jkdata@posteo.de
phone:           +49 341 978 549 41
abuse-c:         ACRO47411-RIPE
mnt-ref:         jkdata-de-mnt
mnt-by:          jkdata-de-mnt
created:         2022-03-31T06:52:15Z
last-modified:   2023-01-04T08:48:00Z
source:          RIPE

“This object is the central starting point for managing data in the RIPE Database. All your other objects are related to this object. If you manage any aspect of any resource then you should have an organisation object so that other people know who you are, what you maintain and how to contact you.” (Source: RIPE, Description of the ORGANISATION Object)

Requesting Internet Number Resources

To actually request Internet number resources, your organisation has to either become a LIR or find a sponsoring LIR that will submit a request on its behalf. Which option fits best depends mainly on your organisations use-case and budget.

Being a LIR requires a RIPE membership, which currently costs an annual fee of EUR 1.550 and a EUR 1.000 setup fee (as of 2023). Commercial offerings for sponsoring LIR services start at about EUR 200 annual fees. The RIPE account and organisation-object that you created earlier can be used to apply for a membership or as an anchor to get sponsored resources assigned to.

See these documents for all the details:

If your organisation has become a LIR itself, request resources through the LIR Portal. Otherwise find an existing LIR that is willing to act as a sponsoring LIR for your resources. This can be a company specialised in LIR services or, for example, a reqional ISP. (See the section on finding transit providers; they can usually act as a sponsoring LIR.)

By the way: Many organisations start by getting Internet number resources assigned through a sponsoring LIR and become a LIR themselves later on, when their business has grown or they want to sub-assign resources on their own.

RIPE nowadays requires some sort of contractual relationship between your organisation and your sponsoring LIR. They have to ensure a few duties and liabilities are met; that there’s someone to contact in case of fire etc. pp.

See these documents in case you want to work with a sponsoring LIR:

Maintaining More RIPE Database Objects

Now that your organisation has got its own ASN and IP prefix(es) assigned, there are database objects that represent these number resources. It’s time to work with and build upon them.

The objects covered from here on (except INETNUM/ INET6NUM objects) serve a more technical purpose, as the information contained with them is commonly used by other network operators (e.g. as a source of truth for safer BGP configuration) and systems like the Domain Name System (DNS).

This is how the resources transfered to me, so far, look on the RIPE database web-interface when logged in:

Sponsored Resources (Both resources are tagged as "Sponsored resource")

AUT-NUM Object

The autonomous system is represented through the AUT-NUM object. Here’s the aut-num object for AS47462 again (to save you from scrolling around):

aut-num:         AS47462
as-name:         JKDATA-DE
org:             ORG-JA512-RIPE
import:          from AS200297 accept ANY
export:          to AS200297 announce AS47462
import:          from AS21413 accept ANY
export:          to AS21413 announce AS47462
sponsoring-org:  ORG-ASG32-RIPE
admin-c:         JKA
tech-c:          JKA
status:          ASSIGNED
mnt-by:          RIPE-NCC-END-MNT
mnt-by:          jkdata-de-mnt
created:         2008-06-23T11:25:16Z
last-modified:   2023-09-19T21:15:16Z
source:          RIPE

The import- and export-statements document the routing policy for your organisations AS and are commonly used by the operators of neighboring ASes: transit ISPs, customers or peers. For example, while connecting your network with the network of a transit ISP, an export-statement serves as a documented proof that it is your organisations intention to route its prefixes over this ISPs network (autonomous system). A corresponding import-statement inside the transit provider’s aut-num object documents the bi-directional nature (agreement) of that relationship.

Remarks are often used to document available BGP-communities and their meanings & impact within an AS.

AS-SET Object

When your AS has multiple peers or downstreams, it is common to reference AS-SET objects instead of seperate AUT-NUM objects to document routing policies. An AS-SET is a list of AUT-NUM objects or even more AS-SETs. See Description of the AS-SET Object (RIPE).

Your ASes upstreams will almost certainly reference AS-SET objects in their AUT-NUM objects. Unless they reference your AS(-SET) (and thus your downstream ASes) in their export-statements, your prefixes (and those of your downstreams) won’t get announced past them when routes are filtered agains IRRDBs. This can be a common pitfall and even bigger ISPs are just cooking with water; Check their objects too if your prefixes don’t show up in the wild.

INETNUM/ INET6NUM object

The INETNUM and INET6NUM object types are from the Internet number registry part of the database. As soon as your organisation gets IP space assigned there will be such objects documenting these assignments. As always, consult the RIPE database documentation for all the details:

Altough these object types do not play a role in terms of BGP or DNS, they are closely associated with objects that do: namely the ROUTE, ROUTE6 and DOMAIN objects types.

Here’s the INET6NUM object for the IPv6 address space I got assigned for JKDATA-DE (2001:67c:2708::/48) so far:

inet6num:        2001:67c:2708::/48
netname:         DE-JKDATA1
country:         DE
org:             ORG-JA512-RIPE
sponsoring-org:  ORG-ASG32-RIPE
admin-c:         JK13907-RIPE
tech-c:          JK13907-RIPE
status:          ASSIGNED PI
mnt-by:          RIPE-NCC-END-MNT
mnt-by:          jkdata-de-mnt
created:         2022-05-31T12:23:29Z
last-modified:   2022-06-09T18:41:45Z
source:          RIPE

Note the “ASSIGNED PI” status; As I’m working with sponsored resources here, my IP space is assigned provider independent. Refer to the Internet Protocol (IP) address ranges section above if you want to recall what that means.

ROUTE/ ROUTE6 Object

The ROUTE and ROUTE6 object types are, broadly speaking, the IRRDP counterparts of the INETNUM and INET6NUM object types. Like AUT-NUM and AS-SET objects, route objects are commonly used by network operators to verfiy that peers are eligible to announce certain resources via BGP.

Take this simple ROUTE6 object for example:

route6:          2001:67c:2708::/48
origin:          AS47462
mnt-by:          jkdata-de-mnt
created:         2022-06-09T18:43:19Z
last-modified:   2022-06-09T18:43:19Z
source:          RIPE

It creates a connection between an IPv6 prefix (2001:67c:2708::/48) and the AS (47462) that is allowed to announce this prefix over BGP. I was able to create this ROUTE6-object because it corresponds to the INET6NUM object from above, which in turn is linked to my organisation because RIPE assigned the corresponding IP space to it. This makes AS47462 the one and only legitimate origin of the prefix 2001:67c:2708::/48.

Remember that IRRDB lookups are by no means part of the Border Gateway Protocol! Other network operators (particularly the bigger ones) are most probably using IRRDBs to verify what your AS signals them over BGP - but it's no necessity.

In a perfect world, BGP route origin validation had prevented Pakistan Telecom (AS17557) from hijacking YouTube traffic on a global scale in 2008. Yet there is this list of public BGP incidents on Wikipedia..

DOMAIN Object

DOMAIN objects are not mandatory but it’s highly recommended to create them for your organisations prefixes. They are used to delegate domain name system (DNS) servers to use for reverse translation of IP addresses from your prefixes.

This allows applications (like the ping or traceroute tools) to resolve domain names from IP addresses. See this example output from a traceroute to one of my edge-routers (I truncated the RTTs for better readability):

# traceroute --icmp 2001:67c:2708:a::1
traceroute to 2001:67c:2708:a::1 (2001:67c:2708:a::1), 30 hops max, 80 byte packets
1  2a03:4000:4b::3 (2a03:4000:4b::3)
2  2a00:11c0:47:3::fa (2a00:11c0:47:3::fa)
3  2a00:11c0:47:1:47::141 (2a00:11c0:47:1:47::141)
4  Te0-0-0-0-pr2.FRA.router.colt.net (2001:7f8::201c:0:2)
5  fe80::2a0:a50f:fc90:c7d2%ens3 (fe80::2a0:a50f:fc90:c7d2%ens3)
6  2001:920:19d0::77d (2001:920:19d0::77d)
7  er61.jkdata.de (2001:67c:2708:a::1)

Where reverse DNS (rDNS) delegations and records are correctly set up, traceroute is able to show the hostname of the responding routers (hops #4 and #7). This makes it easy to spot the names of ISPs or geographical locations, which are often part of the nodes names. (The name of hop #4 contains “FRA” and “colt.net”, suggesting it’s a router from Colt Ltd. located in Frankfurt, Germany.)

Here’s the DOMAIN object for my prefix 2001:67c:2708::/48:

domain:          8.0.7.2.c.7.6.0.1.0.0.2.ip6.arpa
nserver:         dns01.jkdata.de
nserver:         dns02.jkdata.de
admin-c:         JA9284-RIPE
tech-c:          JA9284-RIPE
zone-c:          JA9284-RIPE
mnt-by:          jkdata-de-mnt
created:         2022-11-13T22:18:40Z
last-modified:   2022-11-13T22:18:40Z
source:          RIPE

The nserver-fields register a delegation for the zone 8.0.7.2.c.7.6.0.1.0.0.2 under ip6.arpa to my nameservers (dns01.jkdata.de and dns02.jkdata.de). Only the parent zone for a INETNUM/ INET6NUM object is created as a DOMAIN object in the RIPE database. Further zones or delegations (for subnets) can be maintained via pure, hierarchical DNS mechanisms.

I wrote a guide on how to setup DNS servers for reverse zones:

Connecting to Other Providers

Coming to the more practical aspects of establishing a new ISP..

The Internet is a huge network of networks, inter-connecting about 65% of the worlds population and over 25 billion devices (as of 2023). This means a hell lot of wires, optical fibers, radio waves, pigeons6 and intermediate devices (like switches, routers, radios, wavelength-division multiplexers, repeaters etc. pp). that are required to transport IP traffic nearly everywhere.

Suitable ways for signalling which traffic belongs where and how to get there are crucial on a local, regional and global scale.

Submarine cable map (Submarine cable systems and landing stations. Screenshot of submarinecablemap.com)

The following sections cover how ISPs connect to each other in order to gain reliable, global reachability for their customers and services.

Physically Inter-Connecting Networks

Direct connections between thereby adjacent ISPs are usually made inside datacenter facilities, often referred to as colocations or points of presences (POPs).

Nowadays that almost ever means patching physical connections (in the form of optical fibers) between your Ethernet switches or routers and those of hopefully at least two other network providers.

This is probably the most expensive part so far. Connecting to a transit ISP easily costs about EUR 500 - 1500 monthly for an uncapped 1 Gb/s link - and you’ll need at least two of them for redundancy.

Even if your network won’t need 1 Gb/s bandwidth towards the rest of the Internet, there’s probably not much to safe here as prices do not decrease linearly for narrower links.

There will also be fees for cross connects (or “X-connects” - telco lingo for overpriced patches between your racks and those of available carriers), actual rackspace for your devices and energy.

Finding Transit Providers

There is no mandatory registry that lists which ISPs are present in which facilities. However, many network operators voluntarily maintain records for their POPs in the widely-used peeringdb.com. (See my record as an example.) It’s a valuable source especially for finding mid-sized and regional providers and thus especially potential networks for peering (in contrast to transiting).

The presence of typical tier-1/2 providers like Deutsche Telekom, Lumen, Hurricane Electric etc. pp. can be enquired directly or by the operators of datacenters. Datacenters can be found via databases like datacentermap.com.

If budget’s rather limited, it’s probably your best bet to use your contacts or make new ones. Organisations that are already operating networks and server infrastructure in the colocations you’re considering may be happy to share some rackspace, bandwidth and costs. If it’s their first time selling transit to a downstream network, point them to the right section of this guide. ;-) #fixme

Edge-Port Security

Without going into too much detail - keep in mind that interfaces towards external networks should be treated with caution. Make sure that your equipment won’t leak information about your internal network over protocols like CDP/ ISDP or LLDP. Also make sure not to accept or send STP BPDUs on such interfaces to prevent instababilty due to network topology changes. Same goes for protocols like VTP, DTP and UDLD. Disable auto-negotiation of link speeds in accordance with your peers.

Considering routers, disable IGMP, MLD and RA messages. Interfaces to external networks should be configured as statically as possible to avoid any unforseen bahaviour.

The AMS-IX (Amsterdam) Internet exchange point publishes a useful guide on configuring peering interfaces. It contains configuration examples for many common network equipment vendors. More vendors and operating systems are covered in a similar guide published by the SIX (Seattle) Internet exchange.

Multihoming Your Network With BGP

"Every second, millions of hosts send billions of packets across the Internet to other hosts, with nothing more than the destination IP address to guide them along the way. Internet service providers (ISPs) use the Border Gateway Protocol (BGP) to inform each other which IP address goes where."

"BGP", Iljitsch van Beijnum, O’Reilly, 2002

Most organisations that think about getting their own IP address ranges do so because they want to multihome their network. That means that they want to connect to more than one ISP - usually to increase the reliability of their Internet connection.

“But we already have a backup connection (and we don’t care about BGP for that)” you might think. In this case your organisation is probably not hosting any mission critical services that need to be reachable from outside of its network:

While it is relatively simple to switch over outgoing traffic to a fallback Internet connection in case your primary one fails (many SMB routers know that trick nowadays), keeping your services reachable from outside of your network involves a bit more effort.

This is because every carrier (ISP) serves you with IP addresses from the prefixes that are assigned to it by its regional Internet registry (RIR). Although your routers, firewalls and workloads can use addresses from both7 upstream networks, your systems will be reachable only over the remaining carriers addresses in case one fails. Whenever your customers are trying to reach you over the temporarily unavailable carriers addresses they’ll be out of luck.

         [Carrier-1]                   [Carrier-2]
         192.0.2.0/28                  198.51.100.0/29
         2001:DB8:bf::/56              2001:DB8:c5::/56
                 │                             │
                 │         ┌─────────┐         │
                 └─[RTR1]──┤WORKLOADS├──[RTR2]─┘
                           └─────────┘
         [www1] .1 / ::1                [www1] .1 / ::1
         [www2] .2 / ::2                [www2] .2 / ::2

(Workloads using addresses from different ISPs prefixes.)

This is where your own IP prefixes and the use of BGP can help:

The idea is to give your systems addresses from your own prefixes and tell your transit ISPs to route traffic destined for your network - your autonomous system (AS) - towards your own routers. In case of a major outage at Carrier-1, your systems IP addresses are still reachable over Carrier-2. Your services remain online.

This is called BGP Multihoming:

      [Carrier-1]                     [Carrier-2]
       AS64496                         AS64511
      192.0.2.0/30                    198.51.100.0/29
      2001:DB8:bf::/124               2001:DB8:c5::/64
              │                                   │
              │                                   │
              └─[RTR1]───[YOUR-PREFIXES]───[RTR2]─┘
                  X       AS65551             X
                         203.0.113.0/24
                         2001:DB8:ca::/48
                                │
                           ┌────┴────┐
                           │WORKLOADS│
                           └─────────┘
		            203.0.113.0/29
		            2001:DB8:ca:a::/64
                            [www1] .1 / ::1
		            [www2] .2 / ::2

(Workloads using addresses from your own prefixes.)

AS47462 multihomed stub in the DFZ (AS47462 - a multihomed stub network in the DFZ | Source)

Provided that you receive not just default routes, but full Internet routing tables from your upstreams, the use of BGP will not only help you circumvent carrier outages, but also find the shortest AS-path to destination networks and balance traffic over your transit links to a certain degree.

Establishing BGP Sessions

Once you got links up and running, you establish BGP sessions between your routers and those of your transit providers. You configure BGP to exchange network layer reachability information (NLRI) with your peers.

Tell your transit ISPs to send you full BGP feeds. That means that they will be sending you a full Internet routing table (per protocol - IPv4 and IPv6) from their point of view. When you receive full Internet tables from at least two providers, your own routers will have multiple paths towards most destination networks on the Internet. The BGP best path algorithm will effectively result into different routes to different destinations being installed in your routers forwarding tables, depending on (among other tie-breakers) the number of ASes a packet from your network has to traverse to reach its destination.

Your own routers need to announce your own network (IP prefixes) to your transit ISPs. BGP will then propagate your announcement to their peers and so on - resulting in your network being visible in the routing tables of ISPs all over the world.

Congratulations - your routers no longer depend on a simple default route to reach other networks. Your multihomed stub network is now a part of the default-free zone (DFZ)! Welcome to the Internet!

Verifying BGP Propagation

Whenever you add or remove prefix announcements towards external BGP peers, e.g. through configuration or simply by (dis-)connecting a BGP-router, you potentially affect the routing table of thousands of routers worldwide. Convergence (global route propagation) can take a few minutes. However, when you announce a network, it’s usually present in the routing tables of providers operating in the same country within seconds.

Use a host on an external network (e.g. a VPS hosted at a cloud provider or simply your residential internet access) and ping the addresses of your BGP routers. Once they reply to your echo requests you can assume things are working as desired.

Additional Resources

Footnotes

  1. https://www.demandsage.com/internet-user-statistics/ ↩

  2. Internet service provider Classifications, Wikipedia ↩

  3. IANA IPv4 Special-Purpose Address Registry / IANA IPv6 Special-Purpose Address Registry / Special-Purpose Autonomous System (AS) Numbers ↩

  4. How much IPv6 addresses can you get with a PI assignment? ↩ ↩2

  5. The LIR organisation of the German Federal Republic is called de.government. See IP und ASN Referenzhandbuch der öffentlichen Verwaltung in Deutschland ↩

  6. See A Standard for the Transmission of IP Datagrams on Avian Carriers (RFC1149) ↩

  7. Although this is perfectly possible with IPv6, it regularly introduces pitfalls with IPv4. ↩

[back to top]