You are on page 1of 44

Redbooks Paper

Vic Cross

Linux on IBM

zSeries and S/390:

VSWITCH and VLAN Features of z/VM 4.4
Introduction and overview
z/VM® Virtual Switch (VSWITCH) is a new z/VM networking function announced by IBM® for z/VM Version 4 Release 4. This new feature is designed to improve the interaction between guests running under z/VM and the physical network connected to the zSeries® processor. IBM also announced IEEE 802.1Q VLAN support for z/VM Virtual Switch, z/VM QDIO Guest LAN, and z/VM HiperSockets™ Guest LAN, allowing z/VM guests to create and participate in virtual LAN configurations. This Redpaper describes how the features can be utilized by Linux guests, and how your virtual networking configurations can be greatly simplified through the use of these new functions. The following topics are covered: Introduction to VLANs z/VM Virtual Switch (VSWITCH) Configuring Linux for VLAN Comparing VSWITCH with router-based designs High availability using z/VM Virtual Switch We also describe some of our experiences with using z/VM Virtual Switch and VLANs on z/VM simulated networks.

© Copyright IBM Corp. 2003. All rights reserved.

ibm.com/redbooks

1

Introduction to VLANs
Under z/VM 4.4, VSWITCH and Guest LAN include the capability to support IEEE 802.1Q Virtual LANs within the simulated network. This section introduces the concept of Virtual LANs and some of the terminology involved.

What is a Virtual LAN
A virtual LAN allows a physical network to be divided administratively into separate logical networks. In effect, these logical networks operate as if they are physically independent of each other. The concept can be difficult to describe in words, so Figure 1 shows a diagram to illustrate it.

VLAN11

LINUXA

Hub Switch 1
Router
VLAN10 VLAN12

Switch 2

Figure 1 VLAN scenario

We make the following observations about the network in this diagram: Switch 1 is installed at one location, and Switch 2 and Hub are in another (for example, on separate floors of a building, or in separate buildings). Switch ports are represented by dark squares. The larger squares are trunk ports, the small squares are access ports.

2

Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.4

Definitions: A trunk port is a port that carries VLAN traffic between VLAN-aware devices such as switches. Trunk ports provide the connections that carry traffic for more than one VLAN between the switches that are used to access those VLANs. An access port is a port that is defined to a single VLAN only. Access ports provide connections for non-VLAN-aware devices, giving them access to the VLAN environment. Frames sent on access ports do not have any VLAN information attached. VLAN11 is a network that exists across both locations. This is achieved by defining trunk ports on both switches to the VLAN, and then connecting the two trunk ports. The router is a VLAN-capable device, attached to one of the trunk ports. The correct definitions in the router will provide a routing path between VLAN10 and VLAN12, and between either of these networks and the WAN (represented by the cloud). This is usually done by defining a virtual network device against the physical port on the router, linking that virtual interface to the VLAN, and enabling the interface for routing. VLAN11 has no access to any other VLAN, or the WAN. Even though VLAN11 shares access to trunk ports in both switches with VLAN12, the VLAN architecture prevents traffic from flowing between the two networks. If routing to VLAN11 is required, the Switch 1 trunk port to the router could be included into VLAN11 and a virtual interface for VLAN11 defined in the router. The only machines in the entire network that are permitted to access the server LINUXA are those in VLAN11. This is for the same reason that VLAN11 cannot access the external network: there is no routing path between VLAN11 and the other VLANs. Attention: VLANs are different from Emulated LANs (ELANs). A VLAN uses the same frame format as the underlying medium, while an ELAN often uses one network technology to carry the frames of another. An example of this is Asynchronous Transfer Mode (ATM) ELAN, which allowed Ethernet and Token Ring traffic to be carried over an ATM backbone by emulating those frame formats over ATM cells. Note that VLANs and ELANS should not be confused with the various simulated LAN technologies we use on zSeries and z/VM!

VLAN standards
Several virtual LAN mechanisms exist; most of them are proprietary and operate on a single vendor’s equipment.

Port-based VLANs
Port-based VLANs are most often proprietary solutions that function within a single vendor’s switch hardware. They provide a method of dividing a single network device (the switch) into separate broadcast domains; how the switch does this internally is platform-specific. Importantly, end stations have no way to participate in multiple VLANs because the switch isolates the devices attached to it from the VLANs in the switch.

IEEE 802.1Q VLAN tagging
IEEE 802.1Q defines a standard virtual LAN mechanism that is being implemented across equipment from many different vendors. It uses a header, called the VLAN tag, added to packets transmitted over the network. The tag contains information that allows the network to manage the separation of the different virtual LANs.
Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.4

3

Cisco Inter-Switch Link VLANs
Cisco Systems has a proprietary VLAN trunking mechanism called Inter-Switch Link (ISL). If your site uses Cisco networking equipment, ISL VLANs may be in use. ISL VLANs are not compatible with IEEE 802.1Q VLANs, but Cisco switches provide a function that allows mapping between ISL VLANs and 802.1Q VLANs. Important: In this document we will investigate the IEEE 802.1Q VLAN only, because this is the VLAN standard supported under Linux and z/VM. For the remainder of this paper, when we refer to VLANs, we are specifically referring to IEEE 802.1Q VLANs (unless stated otherwise).

How IEEE 802.1Q VLANs work
VLAN adds additional information to the data packet. The extra data is referred to as the VLAN tag. The tag appears immediately after the Ethernet frame header, and before the data payload of the frame. The format of the VLAN tag on Ethernet is shown in Figure 2.
Octets
(max 36)

1-2

3-4

7

N

TPID
(Ethernet encoded)

TCI

Length/Type field

E-RIF
(present only if required)

Octets 1

2

CFI

priority

VID

Bits

8

6

5

4

1

8

1

Figure 2 VLAN tag format for Ethernet

In most instances, only the Tag Protocol Identifier (TPID) and Tag Control Information (TCI) fields are used. This gives the impression that the VLAN header is only four bytes. In fact, the specification defines that additional information can be carried in the tag, including an Extended Routing Information Field (E-RIF) which would be used in source-routing environments (the Canonical Format Indicator (CFI) bit in the TCI indicates whether the E-RIF is present or not). The three-bit priority field at the start of the TCI can be used by switch equipment to prioritize frames on different VLANs. On Linux, the vconfig command has parameters that allow the priority field to be set for the VLAN being defined.

4

Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.4

16 172.Note: The VLAN priority is separate from IP priority mechanisms. IP Layer Routing 172. Still other prioritization schemes may exist.26.26. The switch can assign a default VLAN number to be assigned to any untagged frames that enter the switch.53.4 5 . like the traffic shaping facilities provided by Linux.8 172. Connecting to a VLAN VLAN-capable devices attach to VLANs through the use of virtual network interfaces.55. or it can tag the frames with a port-specific VLAN number (providing a function similar to a port-based VLAN).26.56. The VLAN tag is never included in a packet sent to a non-VLAN device.16 172.16 Addresses Interface queues vlan20 eth0 eth1 vlan21 vlan30 vlan40 eth2 8021q 8021q NIC driver NIC 1 NIC driver NIC 2 NIC driver NIC 3 Figure 3 Layer diagram for VLAN-capable devices Attention: This diagram is not meant to be a representation of the way that VLANs are implemented in any particular device.0. Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. while IP prioritization operates within the IP layers of routers.26.60. VLAN priority is used to prioritize the frames of a VLAN relative to other VLANs.0 172. The action taken by the switch in this case can vary.26. Each VLAN has its own separate virtual interface. Untagged frames A device does not have to support VLANs in order for it to join a network that uses them. It is a conceptual overview only. A frame generated by a device that is not VLAN-capable is called an untagged frame when it arrives at the VLAN-capable switch.16 Source 0. Part of the function of a VLAN-capable device is to add or remove VLAN tags as required.26.0. usually based on the learned capability of the peer device.59.15.16 172. The diagram in Figure 3 shows the layered relationship of the components involved in packet transmission.

Note: In Linux.0.26. it can act both as a trunk port to handle tagged frames. When using VLANs over Guest LAN. The Guest LAN simulation works in exactly the same manner as in previous z/VM releases.4 . Linux. see “Untagged frames” on page 5 for more on this. the Guest LAN simulation 6 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.0. The IP address associated with eth1 is a dummy address (that is. when VLAN interfaces are configured. as illustrated with eth1 in Figure 3 on page 5. a simulated QDIO or HiperSockets network can now provide IEEE 802. An example of what can happen with untagged frames in a VLAN environment is shown in “VLAN Isolation with untagged frames” on page 32. when the packet gets to the front of the queue it is given to the NIC driver for encoding and transmission on the network with no other action. As for any other interface. This allows the VLAN support to send the packet to the correct interface queue.1Q specification refers to this kind of port as a hybrid port. The use of untagged frames in a VLAN environment can result in unexpected connectivity problems and potential security issues. however. places the packet onto the queue for eth0. on the other hand. resulting in an untagged frame being transmitted. respectively. and as an access port to handle untagged frames. the physical interface the VLAN is associated with must be provided as part of the configuration. This means that switch port does not need any configuration to support untagged frames (it can operate purely as a trunk port).16 as a source address.In Figure 3. To avoid this. eth1 is an interface that supports connection to two VLANs. which is commonly used for this purpose because it can never be used as a source address.1Q VLAN function to attached guests. because the IP layer will not direct any packets to that interface for transmission (nor will the network send any packets to that address). except for the address associated with it. When the IP layer selects 172. 20 and 21. This means that the device does not generate any untagged frames when communicating with VLAN-aware devices. the kernel VLAN code processes the packet by adding the VLAN tag and passing the tagged packet to the driver for the interface that the VLAN belongs to. once a routing decision has selected eth0 as the interface for the packet to be transmitted on. You can do this by using a dummy address on the base interface. The 802. it must be configured to handle these frames. a valid IP address is configured on the eth2 interface. VLAN support on z/VM Guest LAN With z/VM 4. packets may arrive at the queues for these interfaces as a result of the IP layer making a routing decision.4. except that it can pass VLAN tagged frames between guests that are VLAN-aware. Note: Some equipment (such as Cisco switches/routers) specify a default VLAN used for any interfaces that do not have an explicit VLAN ID coded. The practical effect of this is that the IP interface eth1 will not generate any traffic. The switch that eth2 is connected to must know what to do with such frames arriving when tagged frames are expected.0. Figure 3 shows eth1 configured as 0. always ensure that network traffic specifies a VLAN ID. eth2 has an almost identical configuration to eth1. Because eth0 is non-VLAN.53. The IP layer. If the switch that this interface connects to is VLAN-aware. Here. an address which does not match a real network configuration). The virtual interfaces for these VLANs are vlan20 and vlan21. eth0 is a non-VLAN device. can generate untagged frames in the way described for eth2. the packet is not processed by VLAN. For VLAN interfaces.

this is not a limitation. This allows you to configure your guests with IP addresses from the network that the OSA Express connects to. but this is not the case. in addition to the IP multicast support added in a previous release.does not interfere with the VLAN tagging of network frames.1Q VLAN support We describe various functions provided by the new support in the following sections. This means that a guest attached to a Guest LAN must be VLAN-aware in order to participate in the VLAN network. VSWITCH provides a function that operates much like a Layer 2 switch within z/VM. Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. Important: Avoid using the term “bridging” to describe this function. To z/VM guests. bridging usually refers to the copying of an entire Layer 2 frame from one network to another (performing whatever frame translation is required on the way. such as in the case of translational bridging between Token Ring and Ethernet). Because QDIO is an IP-only transport anyway. Guests attached to the VSWITCH appear to be attached to the LAN that the OSA Express is attached to.4 is broadcast capability. We mention this mainly so that you don’t incorrectly refer to VSWITCH as a bridge when talking to the network staff. where either an OSA port is not associated. z/VM Virtual Switch can also function in a disconnected mode. and manages the operation of the OSA Express adapter ports the VSWITCH uses. External network access using VSWITCH VSWITCH provides a way to link an external network to guests under z/VM via an OSA Express. it operates almost exactly the same as a z/VM 4. z/VM Virtual Switch (VSWITCH) z/VM Virtual Switch is an extension of the Guest LAN simulated networking function provided in earlier releases of z/VM. the HiperSockets microcode on zSeries processors does not support VLAN-tagged frames. Important: At this time. without the need for a routing function. TCPIP service machine “controllers” z/VM Virtual Switch uses a TCPIP service machine to act as the interface to an OSA Express network connection. It might seem that a VSWITCH without an OSA is just the same as a QDIO Guest LAN. without the need to configure Proxy ARP in a z/VM TCP/IP service machine. This means that guests attached to a simulated HiperSockets can now use applications and protocols that use IP broadcast.3 QDIO Guest LAN. the VSWITCH provides additional control over VLAN membership and handling of untagged frames (refer to “VLANs on z/VM Virtual Switch” on page 9 for more detail). This TCPIP service machine acts as a controller for the VSWITCH. and as such does not qualify as a full Layer 2 network bridge. or the associated OSA does not flow traffic to the external network. VSWITCH will handle only IP packets.4 7 . Another feature added to z/VM simulated HiperSockets with z/VM 4. with two important exceptions: Direct external network access via OSA Express IEEE 802. While this function looks like it bridges between a Guest LAN and the Ethernet. VLAN support applies only to simulated HiperSockets on z/VM 4.4.

the *VSWITCH system service scans its list of eligible TCPIP service machines for the one with the fewest current VSWITCH connections and selects that service machine as the controller. you can see the device and link entries created for the VSWITCH in the output of the NETSTAT DEVLINKS command. Example 2 NETSTAT DEVLINKS output on a TCPIP VSWITCH Controller netstat devlinks VM TCP/IP Netstat Level 440 Device VSWITCHDEV Queue size: 0 CPU: 0 Link VSWITCHLINK BytesIn: 4355 Device TESTSW32320DEV Queue size: 0 CPU: 0 Type: VSWITCH-IUCV Status: Connected IUCVid: *VSWITCH Priority: B Type: IUCV Net number: 1 BytesOut: 4489 Type: VSWITCH-OSD Address: 2320 Status: Ready Port name: OSA2320 8 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. Using the CP QUERY CONTROLLER command. Example 2 shows the results of this command. that stack will act as the controller. or when an OSA RDEV is added for an existing VSWITCH.4 . you can see the TCPIP stacks that can serve as VSWITCH controllers.01/0. and that TCPIP service machine is eligible to act as a controller. the *VSWITCH system service determines which TCPIP service machine is to be used as the controller for that OSA. at least one TCPIP service machine must be configured to be a controller (this configuration is described in “Planning for VSWITCH” on page 10). In order for a VSWITCH to provide connectivity to a LAN. If no CONTROLLER parameter was given. Example 1 CP QUERY CONTROLLER q controller CONTROLLER TCPCTL1 Available: YES SYSTEM TESTSW3 Controller: * CONTROLLER TCPCTL2 Available: YES Ready. or CONTROLLER * was specified. otherwise. The configuration allows the TCPIP stack to connect to the system service that manages VSWITCH connections. T=0. Once the controller stack for a VSWITCH has been selected.01 17:12:16 VDEV Range: * VDEV Range: * When a VSWITCH with an OSA RDEV (real device) is defined. The TCPIP service machine is then able to act as a controller for a VSWITCH OSA connection. Example 1 shows the output from the command. At least one controller TCPIP service machine must be eligible. The *VSWITCH system service dynamically adds the required definitions to the controller.Important: The controller TCP/IP service machine is not involved in data transfer between the LAN and the guests on the VSWITCH. When a TCPIP service machine is controlling an OSA Express port for a VSWITCH. Important: You must not define the OSA Express port to your controller TCPIP service machine in its PROFILE TCPIP. the *VSWITCH system service instructs it to activate the OSA port specified using the configuration given in the CP DEFINE VSWITCH or CP SET VSWITCH commands. If a CONTROLLER was specified on the CP DEFINE VSWITCH or CP SET VSWITCH command. the request to add the OSA to the VSWITCH will fail.

Note: This means that the guest does not require any configuration for VLAN support. T=0. This is the connection to the *VSWITCH system service. and is described in “VSWITCH VLAN filtering”. VLANs on z/VM Virtual Switch z/VM Virtual Switch supports IEEE 802. VSWITCH can provide many options for controlling which guests receive frames on which VLANs.01/0. Frames are transferred directly between the OSA Express buffers and the guest on the VSWITCH. Likewise. CP will act as a VLAN-aware switch and perform VLAN tag processing according to configuration. created when the stack initializes with VSWITCH CONTROLLER ON specified in PROFILE TCPIP. You do not need to set PRIROUTER on a VSWITCH for normal operation. When an OSA Express is attached to the VSWITCH. the frames will be sent and received untagged. This name is derived from the name of our VSWITCH (TESTSW3) and the virtual device number of the OSA. Guests that do not support VLAN will send and receive frames on the VLAN that CP has been configured to connect them to.01 10:31:55 Arp Query Support: Yes Type: QDIOETHERNET Net number: 0 Yes Yes Here. In the case where the guest is linked to a VLAN via VSWITCH configuration. This means that guests attached to a VSWITCH under z/VM 4. As long as you do not set PRIROUTER on your VSWITCH definition. There are additional considerations for OSA Express port sharing when multiple OSA Express ports are used for backup. On a VSWITCH. Sharing an OSA Express port with z/VM Virtual Switch The OSA Express port you use with your z/VM Virtual Switch can be shared between the VSWITCH and other guests (or LPARs). VSWITCH will add the correct VLAN tag to frames as they leave the guest and remove it when the guest receives frames. The only time to set PRIROUTER for a VSWITCH is if you have a guest attached to the VSWITCH that is providing a routing function for systems attached to another network. called in this case TESTSW32320DEV.4 can participate in VLAN networking.4 9 . Network data frames destined for guests attached to a VSWITCH are handled entirely by the CP VSWITCH buffer management logic. VLAN operations extend between the VSWITCH and the LAN via the OSA Express. The link type shown here is VSWITCH-IUCV. or even with other VSWITCHes.Router Type: NonRouter Link TESTSW32320LINK Broadcast Capability: Multicast Capability: Ready. outgoing frames are transferred by the VSWITCH code from the guest’s virtual NIC to the OSA Express. See “Multiple OSA Express ports” on page 21 for more information. This is called VLAN filtering. This means that the guest need not be VLAN-aware in order to communicate to a specific VLAN. If the configuration of the VSWITCH does not specify a VLAN ID for that guest. Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. The OSA Express port device.1Q VLANs. you can see two extra devices added that support the VSWITCH: An IUCV link called VSWITCHDEV. you can share it with other images without restriction (even if one of the systems you are sharing with requires PRIROUTER). The controller TCP/IP service machine is not involved in this data transfer.

If an OSA Express is associated with your VSWITCH. you must perform several tasks. Attention: There appears to be one exception to this: regardless of the VLAN IDs specified. You can nominate the VLAN IDs that a guest is allowed to “see” when you use the CP SET VSWITCH command to grant access to a VSWITCH. Planning for VSWITCH Before you can start using a VSWITCH. you can control VLAN access at the VSWITCH. this OSA Express port will function like a trunk port with access to all of the VLAN IDs appearing in the VSWITCH. This could result in unauthorized access to different network resources. If you are using VSWITCH only as a Guest LAN with VLAN filtering. We recommend that you do not use untagged frames in your environment. use the CP SET VSWITCH command to grant the guest access to one VLAN ID only. The rules that apply to packet delivery on a z/VM Virtual Switch are explained in the “Working with Virtual Networks” chapter of z/VM Virtual Machine Operation. SC24-5955. Using z/VM Virtual Switch In this section we describe how we used VSWITCH on our system.4 . Using the CP SET VSWITCH command. Only frames tagged with those VLAN IDs are passed to the guest. For VLAN-unaware guests. the guests on the VSWITCH will now be present on the same VLAN. 10 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. you can choose to make that port belong to any of the VLAN numbers configured in the VSWITCH. For VLAN capable systems. untagged frames from within the VSWITCH are always passed to the guest if the destination IP address matches an address that the guest has configured. Only a virtual adapter “attached” to the correct VLAN will receive packets on that VLAN. We describe the planning and configuration steps we took to enable VLAN communication both within our z/VM system and to VLANs outside it. always configure the base interface with a dummy IP address as described in “Connecting to a VLAN” on page 5. VLAN isolation VLAN-capable devices automatically manage the separation of traffic between VLANs. Important: The tasks described here are required only if you are using VSWITCH to connect to an external network via OSA Express. This will extend those VLANs out of the VSWITCH into the LAN. there are no preparation tasks and you can go straight to “Configuring VSWITCH” on page 11. VSWITCH VLAN filtering VSWITCH provides an additional feature that provides further isolation. When you configure the port on the LAN switch the OSA Express connects to. if the VLAN IDs already exist in the LAN. This is analogous to defining the VLAN membership of a trunk port on a LAN switch.

Important: PORTNAME (and its operands) must be the last parameter specified on the DEFINE VSWITCH command (this applies to the SET VSWITCH command also). The following shows the definition of a z/VM Virtual Switch called TESTSW3. You can specify up to three addresses at this parameter. This authorizes the TCP/IP service machine to connect to the *VSWITCH system service. This works exactly the same way as it does for Guest LAN.4 11 . Only the first device address from the OSA port’s address triple is entered here. This is used to provide backup OSA Express connections in case the first OSA fails. you can start using VSWITCH. however: Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.01/0. The real device address of the OSA Express adapter to be associated with this VSWITCH. define vswitch testsw3 rdev 2320 2180 portname osa2320 osa2180 VSWITCH SYSTEM TESTSW3 is created Ready. There is an extra step involved with VSWITCH. This name is used to identify this VSWITCH in later commands. The special value NONE is used to create a Virtual Switch that is not associated with an OSA Express connection. HCPSWU2830I TCPCTL1 is VSWITCH controller. PORTname portname The port name to be used when opening the OSA Express adapter. all LPARs or guests must use the same name to open the OSA port. Configuring VSWITCH Once your z/VM system is prepared. T=0. you must specify more than one portname. The key parameters (the ones you are most likely to use) of the command are as follows: switchname RDEV rdev This is the name of the VSWITCH being created. If you are specifying more than one RDEV. Connecting a guest to VSWITCH Guests connect to a VSWITCH using a simulated QDIO NIC. The full syntax of the command is explained in “DEFINE VSWITCH” on page 39.Update the directory entry for CONTROLLER TCPIPs Any TCP/IP service machine that you want to operate as a controller for a VSWITCH must have the IUCV *VSWITCH statement added to its directory entry. the VSWITCH CONTROLLER statement must appear in the PROFILE TCPIP for that stack. Add VSWITCH CONTROLLER to PROFILE TCPIP For a z/VM TCP/IP service machine to initialize as a candidate to control an OSA for a VSWITCH. Define the VSWITCH A VSWITCH is created using the CP DEFINE VSWITCH command from a z/VM Class B userid. This works the same way as for normal use of the OSA Express: when sharing the OSA. with two OSA Express ports available.01 20:13:02 HCPSWU2830I VSWITCH SYSTEM TESTSW3 status is ready.

devices 6100-6102 defined Ready. 12 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. untagged frames will still be delivered to that guest.01/0. Refer to the section on creating a simulated NIC in Linux on IBM ^ zSeries and S/390: Large Scale Linux Deployment. you can IPL your Linux installation. Also. couple 6100 to system testsw3 Configuring Linux for the VSWITCH connection Once your userid has a simulated NIC defined and connected to the VSWITCH. Important: When a guest’s access to the VSWITCH is restricted to a single VLAN.01/0. SC24-6036. an excellent new chapter called “Working with Virtual Networks” is available in z/VM Version 4. set vswitch testsw3 grant lnxsu6 vlan any Command complete Ready. T=0. This example shows the use of the CP DEFINE NIC command to define a simulated QDIO NIC at device address 6100. Here is an example of using the command to connect the simulated NIC at device address 6100 to a VSWITCH called TESTSW3.01/0. Configuring your Linux system for attachment to a VSWITCH is done in the same way as for a QDIO Guest LAN connection.01 14:38:01 set vswitch testsw3 grant lnxrh2 vlan 201 Command complete Ready. or allow the guest access to any VLAN (the default). T=0. Defining a simulated NIC A simulated NIC is defined on a guest using the CP DEFINE NIC command.4 . Refer to “VSWITCH VLAN filtering” on page 10 for information on how VSWITCH controls frame flow.guests attaching to a VSWITCH must be granted access to the VSWITCH using the SET VSWITCH command.4 Virtual Machine Operation. T=0. Granting access to a VSWITCH using SET VSWITCH Access to the VSWITCH is controlled by the GRANT and REVOKE options of the CP SET VSWITCH command. Note: A simulated NIC of type QDIO can be connected to either a QDIO Guest LAN or a VSWITCH.01 14:38:10 The command syntax of the CP SET VSWITCH command is shown in “SET VSWITCH” on page 39.01 13:46:11 Attaching the simulated NIC to the VSWITCH Once your simulated QDIO NIC is created. The GRANT option allows you to restrict the guest to particular VLANs. define nic 6100 qdio NIC 6100 is created. SG24-6824 for more information on how to define a simulated NIC. connect your simulated NIC using the CP COUPLE command.

to configure the VLAN virtual interfaces. The options available on the DIRMAINT NICDEF command are shown in “DIRMAINT NICDEF” on page 40.1Q VLAN patches. This would be useful if you are using a protocol that relies on the MAC address of a host. This allows the MAC address allocated to a simulated NIC to be defined statically. you need the following: Linux kernel 2. Most relevant to Linux guests is the MACID parameter. the first six are set using the new MACPREFIX option on the VMLAN statement in SYSTEM CONFIG. MACID specifies the last six hexadecimal digits of the MAC address. HiperSockets Guest LANs. First we describe the steps that take place to start a VLAN interface. VLAN configuration process on Linux To connect to VLANs from a Linux system. On a VSWITCH. whether on VSWITCH.1Q VLAN module. Note: On our SuSE systems. NICDEF provides some additional options not available when you use the SPECIAL directory control statement. Activate the physical interface. A user-space utility called vconfig. there is no VLAN configuration required. Configuring Linux for VLAN This section describes the process to start using VLANs in your Linux systems. such as DHCP.o was present). or an earlier kernel with the IEEE 802. The steps involved in starting a connection to a VLAN from a Linux guest are: Configure the VLAN. NICDEF directory control statement NICDEF is a new directory control statement added to z/VM Directory Maintenance (DirMaint) with z/VM 4. then we show you how you can do the steps automatically using the VLAN support included in SuSE SLES8.4. Load the IEEE 802.14 or higher. you do need to ensure that your guest has authority to use the VLAN you intend to use.4 13 . or OSA Express adapters that support VLAN.4. but the vconfig program was available in a prepared RPM package that came with the distribution (the vlan package). Bring up the VLAN interface. rather than dynamically defined by CP when the NIC is created. Configure the VLAN If your VLAN configuration exists entirely within a VSWITCH or a z/VM simulated HiperSockets. Use the CP QUERY VSWITCH command to check VLAN authority.Note: The IBM Redbook Linux on IBM ^ zSeries and S/390: Large Scale Linux Deployment. however. and the CP SET VSWITCH command to change the guest authorization.1Q VLANs (the module 8021q. Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. Configure a VLAN interface. not only was the kernel already prepared to support 802. It provides a new way to define simulated NICs as part of a guest’s directory definition. SG24-6824 gives a description of how to set up Linux for Guest LAN attachment (in the section “Configuring a VM Guest LAN in a Linux guest”).

) character in the interface name.o. you can only change this by first revoking the current authority and granting the new authority. either directly or through a VSWITCH. have a problem with the period (. Even when we used a dummy address. the vconfig command allows you to control the operation of the VLAN interfaces. Use the insmod or modprobe command to load the module before attempting any operations to configure or use VLANs.0. however.1Q VLAN Support v1.1Q VLAN module VLAN support is usually built as a module called 8021q. you will see the following messages in your system log or dmesg output: 802. You can also change the default way in which the virtual devices are named.o module uses the DEV_PLUS_VID_NO_PAD format.0. the interface simply gives you a connection to the network. On SuSE SLES 8. Miller <davem@redhat. changing the relative priority of packets for different VLANs. If you will be connecting to a VLAN in your switch hardware. or you need to set or remove VLAN ANY authority. the 8021q.0.0. ensure that the switch configuration supports the VLAN you intend to connect to.4 . and al your IP addresses and network traffic will be associated with VLANs) you configure a dummy’ IP address on the physical interface. You can specify the priority of packets on your VLANs. we used the ifcfg-ethX file in /etc/sysconfig/network to define the address for our base interface. Configure a VLAN interface You create your VLAN virtual interface using the vconfig command. 14 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.7 Ben Greear <greearb@candelatech. When the module is loaded. so either the VLAN_PLUS_VID or VLAN_PLUS_VID_NO_PAD formats must be used. the network configuration scripts worked correctly.com> All bugs added by David S. The following shows how to load the module using the modprobe command: # modprobe 8021q You do not need to load a module if your kernel includes the VLAN support in the kernel (rather than in a module). If you are not going to be using the physical interface to attach to an IP network (that is. you can configure the interface in the same way as usual. If you are going to use a default VLAN or generate untagged frames.0. Refer to “Switch configuration” on page 24 for aspects of switch configuration that you may need to investigate.0 as a valid dummy address (in fact. and our configuration was very easy). Load the IEEE 802.com> Activate the physical interface The physical interface used to connect to a VLAN must be activated prior to configuring any VLAN interfaces. By default. Apart from creating and deleting virtual interfaces. the IP configuration scripts seemed to recognize what we were doing when we used 0.Important: If the guest is already authorized to a VLAN and you need to change to a different VLAN. You can use the address 0. Some programs.

In addition.Important: One program that has a problem with the dot in the interface name is iptables. so as long as you can use the right names for your VLAN interfaces. The following command will create a virtual interface for VLAN 10 against the eth0 interface: # vconfig add eth0 10 If the default VLAN name format of DEV_PLUS_VID_NO_PAD is set. you can use the ip command to configure the interface instead.29/24 dev eth0.10.168. however.255. The iptables program is used to set up firewall and network address functions in your Linux system.0 up If the iproute2 utility is installed on your system. the system will set up your VLAN interfaces for you. Many distributors now include it by default.29: # ifconfig eth0.4 15 .255. You configure the virtual interface in the same way as any other network interface on your system.10 192. The following command will configure the VLAN 10 virtual interface on eth0 with the IP address 192. and policy routing. the interface to the netfilter code in the Linux kernel. If you plan to use any iptables commands against your VLAN interfaces. and have rewritten their network configuration scripts using iproute2 instead of the traditional IP configuration commands such as route and ifconfig. iproute2 provides the facilities to control Linux advanced routing features like multiple route tables. traffic shaping. SuSE has set up their network scripts. which provides the ip command. configuring VLANs has been a manual process because the network configuration scripts provided with most distributions were not “VLAN-aware”. Tip: The vconfig usage text (shown in “vconfig” on page 40) shows you what your VLAN interface names will look like for different settings of the VLAN name format.29 netmask 255.10. allows a great degree of flexibility in configuring the Linux IP stack. Configuring VLANs on SuSE SLES8 Since VLAN support is a fairly recent addition to Linux.168. You would use these ip commands to get the same result as the ifconfig command above: # ip addr add 192.10 up Tip: The iproute2 utility by Alexey Kuznetsov.10. Bring up the VLAN interface The VLAN interface added using vconfig supports the usual interface configuration commands such as ip and ifconfig. Refer to “vconfig” on page 40 for an explanation of the vconfig command syntax.10 # ip link set eth0.10. Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. you will not be able to use the default name format. the command would create a VLAN virtual interface called eth0.168.

0' BROADCAST='192.255. you will not be able to use SuSE’s configuration method and will have to create your own. the configuration requires that the vconfig per_kernel option be set to on. you will have to update the script to show the correct path to the vconfig program. if you are configuring VLAN ID 10. the path to the vconfig executable may be incorrectly specified as /sbin/vconfig instead of /usr/sbin/vconfig. is not correct). create a file in the /etc/sysconfig/network directory called ifcfg-vlanX. this allows you to add your VLAN interfaces to your network configuration and have them activated automatically at boot-time. This will cause the script to fail. Example 3 ifcfg-vlan example ETHERDEVICE='eth4' BOOTPROTO='static' STARTMODE='onboot' IPADDR='192.201. As an example.201. The following command will start the vlan10 interface: # ifup vlan10 16 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.168. rather than just per interface. The contents of this file should be similar to Example 3. Fixing the VLAN script Unfortunately you may find a bug in the script used to configure the VLAN interface.201.168. which has to match the last part of the ifcfg file name.Restriction: In addition to the naming issue. replacing X with the VLAN number you are configuring. This is used to instruct vconfig which physical interface to configure the VLAN on.0' NETWORK='192.1q script. In the /etc/sysconfig/network/scripts/ifup-802. the details shown in Example 3 would be contained in a file called /etc/sysconfig/network/ifcfg-vlan10. Until SuSE has a fix for this.168.255. you start your VLAN interface by issuing the ifup command with the VLAN interface name as the parameter. the SuSE network configuration processing will not call the correct script to configure the interface. Once you have prepared your ifcfg file. Important: The name of the ifcfg file you create is critical.109' NETMASK='255.4 . If you need the ability to have the same VLAN numbers on different real interfaces representing different VLANs. If the file is named incorrectly (or more accurately: if the name you pass to the ifup script.255' The ETHERDEVICE field is additional for VLAN configurations. so it is unlikely that this restriction will cause a problem for you. The STARTMODE field works the same as for physical interfaces. This means that VLAN numbers have scope across the kernel. VLAN numbers should be allocated universally. Configuring your VLANs To configure a VLAN on SLES8.

240 ms Note: In “z/VM Virtual Switch failover” on page 28. you will have to write a script that implements the steps outlined in “VLAN configuration process on Linux” on page 13. the system network initialization scripts will do all the VLAN configuration for you at system startup. Utilities The ping command is a simple command you can use to verify connectivity. you can now reach LAN devices on that VLAN as well.109).109 traceroute to 192.4 17 . Testing the configuration Once your virtual interface is up. Refer to the discussion there for more details. If you included “STARTMODE=onboot” in the ifcfg-vlanX file for your VLAN interface. Making your VLAN configuration persistent If you use the supplied configuration processes on SuSE SLES8 (discussed in “Configuring VLANs on SuSE SLES8” on page 15). restricting that guest to generating frames for its authorized VLAN only1. we use the -R switch on the ping command to display the route taken both to and from the destination host. Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.098 ms 1.168. if a guest authorized to connect to only one VLAN generates untagged frames.677 ms 0. If your VSWITCH is linked through an OSA to stations on the LAN. See “Activate the physical interface” on page 14 for information on this. you can use it to communicate with other guests on the same VLAN.200.168.026 ms 1.109 1. If you are not using SuSE SLES8.805 ms 0. 30 hops max.200. For example. 40 byte packets 1 192.1 0.201.168.200. you can verify that your VLAN configuration is working as you expect. a utility that will show you the path to other devices in the network.168.Important: The physical interface (the interface specified in the ETHERDEVICE parameter in the interface configuration file) must be activated in order for this to function. Example 4 traceroute command example # traceroute 192. two guests on the same VSWITCH but configured on separate VLANs can only reach each other via a router. In fact.200. This can provide a little more information than traceroute. Using traceroute. In VSWITCH. An example traceroute command execution is shown in Example 4. the VSWITCH applies this VLAN tag to all the frames the guest generates. Comparing VLANs on VSWITCH and Guest LAN Configuring Linux guest connections to VLANs on a z/VM Guest LAN is identical to the process used on VSWITCH. and using traceroute will help you verify this.762 ms 2 192. You can also use traceroute. The difference between Guest LAN and VSWITCH is in the handling of untagged frames.168. you do not need to do anything else to have your VLAN connections start at reboot time. and your LAN is configured with the same VLAN. or you cannot use SuSE’s process because of your configuration requirements.109 (192. the VSWITCH tags them with the VLAN ID the guest is authorized to.

No VLAN filtering takes place.42. 18 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.168.4 . sA .12 sC . Guests that are VLAN-aware must be configured for matching VLAN IDs in order to communicate on their VLAN.8 .13 sD Master VRRP Virtual IP: 192. REDP3657.11 sB . and dynamic routing facilities. and both VLAN-aware and non-VLAN-aware guests will be able to process the packets (as long as the VLAN-aware guests have a network definition that supports untagged frames). Non-VLAN-aware guests will not be able to process these frames. This configuration provides a high level of availability through its use of multiple router guests. will give you more information about this configuration. Comparing VSWITCH with router-based designs Figure 4 shows a virtual network configuration based on Linux virtual routers. Guests with no VLAN support generating untagged frames will not have their frames modified either.1 Guest LAN 192.The z/VM Guest LAN does not process or modify VLAN tags in any way. the Guest LAN will carry those frames unaltered and other guests that are VLAN-aware will be able to process them.9 rA z/VM OSA OSPF rB OSA Switch Switch Router Router Figure 4 Connectivity based on Linux routers 1 This is similar to the configuration of a “default VLAN” on some LAN switches.0/24 Backup . Note: The Redpaper Linux on IBM ^ zSeries and S/390: Virtual Router Redundancy Protocol on VM Guest LANs. Frame processing for VLAN-aware and non-VLAN-aware guests occurs as follows: When guests that are VLAN-aware put tagged frames onto the Guest LAN.10 . multiple OSA ports.168.42.

Also. the function of providing network availability is moved into network components.13 sD VSWITCH 192. Running the networking protocols that provide the routing failover (VRRP.42. Note: In Figure 5.10 . we show the two network routers participating in VRRP to provide a redundant gateway service for the Linux guests. OSA ports must be activated with the PRIROUTER setting. Other such protocols. you remove the overhead of your Linux router guests. may be just as effective in your environment. it does not have to be VRRP. The Linux guests consume memory. it has a few drawbacks in a z/VM environment. a highly available network configuration can be made without the resource overhead of virtual routers.11 sB . Figure 5 shows an example configuration based on VSWITCH that provides a high degree of availability. sA .12 sC . disk and CPU resource just by being active. as well as the network traffic generated by the protocols used to provide high availability in the network. This stops you from sharing an OSA ports between multiple routers.168.168.42.0/24 z/VM TCPIP1 OSA TCPIP2 OSA Switch Switch Master VRRP Virtual IP: 192.While this configuration provides excellent availability for the Linux guests. like Cisco Hot Standby Router Protocol (HSRP). Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.1 Backup Router Router Figure 5 Connectivity based on VSWITCH By using VSWITCH in this configuration. While we strongly recommend that you use some kind of redundant gateway facility. not shown here is that the routers will be participating in the dynamic routing environment used for the rest of the LAN. OSPF) generates network traffic and consumes processor resource. With VSWITCH. where it can be more consistently managed with the rest of the enterprise network function.4 19 . Importantly.

to-point connections to a single shared-access transport facility. All your guests will connect to the VSWITCH in the same way as the Guest LAN they connect to now. you will need to add appropriate CP SET VSWITCH commands to give your guests authority to connect to the VSWITCH. For example. if your VSWITCH connects to an OSA Express. Important: For all migrations to z/VM Virtual Switch. remember to verify that the guest’s routing definitions are correct after the change is made.Migrating to z/VM Virtual Switch Changing your current guest networking configuration over to z/VM Virtual Switch can be achieved with a minimum of reconfiguration effort. you simply change your existing CP SET LAN commands into CP SET VSWITCH commands. If your current Guest LAN is defined as UNRESTRICTED. switch the guest’s NIC definition to QDIO. You cannot define a transient z/VM Virtual Switch. QDIO Guest LAN to VSWITCH If you are currently using persistent QDIO Guest LANs in your z/VM environment. Restriction: If you use transient Guest LANs in your current configuration (LANs defined and owned by z/VM general users instead of SYSTEM). and complete the configuration of the new VSWITCH interface while the HiperSockets Guest LAN is still available. 20 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. vCTC or IUCV to z/VM Virtual Switch Migrating either vCTC or IUCV topologies to z/VM Virtual Switch is a complicated process. the default route for the guests should specify a LAN router rather than a z/VM TCP/IP service machine or Linux router guest. you can simply replace them with a z/VM Virtual Switch. you can prepare new configuration files in advance. and the CP commands that define and modify them can be run by Class B users only.4 . For this migration. they are all owned by SYSTEM. and copy the new configuration files into place from the console. Alternatively (if you’re comfortable with the Linux console). because you cannot use the same simulated NICs for HiperSockets Guest LAN and VSWITCH. we recommend setting up the new VSWITCH in parallel with the existing HiperSockets Guest LAN. If your current Guest LAN is RESTRICTED. because the topology of the guest connectivity changes from a set of point. you will need to consider changing to persistent LANs. This change will require a corresponding change in the guest’s TCP/IP configuration. HiperSockets Guest LAN to z/VM Virtual Switch This migration is a little more complex. Define the new simulated NIC to the guest. You will need to change the NIC definition for the guest from TYPE HIPER to TYPE QDIO. Setting up the VSWITCH and simulated NICs in advance is definitely recommended for this migration.

We describe a way of doing this in “Configuring controller TCP/IP service machines” on page 22. NICDEF provides additional parameters to define simulated NICs. all your external network connectivity via a VSWITCH fails also. with little configuration or management overhead. and multiple TCPIP controller service machines for some software redundancy. Tip: z/VM TCP/IP stacks used as controllers do not need to have any connectivity defined of their own. When you have more than one TCP/IP service machine running in your z/VM system. They do not need DEVICE and LINK statements. As long as your LAN switch is configured appropriately. redundant OSA ports are made available to your VSWITCH in case network access through the first port is impaired. it is highly recommended to have at least one additional TCPIP service machine configured as a controller to provide continued connectivity. This means that you can set up additional TCPIP service machines with no specific IP configuration to serve as backup controllers for your VSWITCH configuration. these stacks can all be set up to become a VSWITCH controller. Example 6 on page 24 shows the output from the command. consider using the new NICDEF statement instead. you can see the TCPIP stacks that can serve as VSWITCH controllers. Planning redundant configuration for VSWITCH Some planning is required to ensure that your VSWITCH connectivity will survive outages. You can define multiple OSA Express adapters for hardware redundancy. High availability using z/VM Virtual Switch z/VM Virtual Switch can be used to create highly-available connectivity for your Linux guests under z/VM. Up to three RDEV values can be Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. or the like. and shows that our system currently has three TCPIP service machines ready to operate as controllers. You can specify multiple OSA Express ports using extra values on the RDEV parameter to the CP DEFINE VSWITCH and CP SET VSWITCH commands. Using the CP QUERY CONTROLLER command. you can ensure that your z/VM guests stay linked to the external network when failures occur. or HOME addresses. You can make access to your z/VM guests highly reliable by configuring the redundancy features of VSWITCH combined with LAN-based high availability. Multiple CONTROLLER TCP/IP service machines If you have only one TCPIP service machine and it is logged off or fails. For this reason.4 21 .Virtual NIC definition If you currently define your guests’ Guest LAN connections using the SPECIAL directory control statement. Multiple OSA Express ports When you specify more than one RDEV on the CP DEFINE VSWITCH command. The syntax of the DIRMAINT NICDEF command is shown in “DIRMAINT NICDEF” on page 40. including: Specifying the virtual channel path identifier (CHPID) the simulated NIC will use (most useful for simulated HiperSockets NICs for use on z/OS guests) Specifying the MAC address for the simulated NIC NICDEF is discussed in “NICDEF directory control statement” on page 13.

This configuration would allow us to easily and quickly start additional controller machines from the same configuration. We also defined two extra TCPIP service machines. the port cannot be used for VSWITCH backup. This means that your backup ports can be in use on other guests.4 . If you don’t do this. if you are extending VLANs between your VSWITCH and the LAN. In our lab configuration. some or all of your guests may fail to connect properly if your main OSA Express connection fails. and the detail of how it was set up appears in the following paragraphs. Demonstration: “z/VM Virtual Switch failover” on page 28 shows the results of the testing we performed with these availability configurations. A very important configuration requirement when defining multiple OSA Express ports for backup is that the switch configuration of the two ports must match. If the primary system using the OSA Express port requires a certain configuration in the switch. Configuring controller TCP/IP service machines You can use your main TCPIP service machine as a VSWITCH controller. They each had their own 191 disk. meaning up to three OSA ports can be provided to connect the VSWITCH to the LAN. which they link to as their 198 configuration disk. if the PRIROUTER setting appears in the port configuration for both your VSWITCH and that of the guest or VSWITCH using the port. As discussed in “Planning redundant configuration for VSWITCH” on page 21. In the following paragraphs we will describe how to build it. you define multiple controller TCPIP service machines and provide multiple OSA Express ports to your VSWITCH. However. allocated as TCPMAINT 298 (the same size as TCPMAINT 198). 22 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. Configuring high availability with VSWITCH Figure 5 on page 19 shows an example of a high availability configuration. The configuration we tried was a success. The second OSA was the one used for direct access by other Linux guests to the LAN switch. Importantly. but we decided to try setting up a separate configuration for a dedicated controller service machine. named TCPCTL1 and TCPCTL2. Important: This may also prevent you from sharing an OSA Express port for backup purposes. Minidisk configuration The controller service machines were given a separate configuration disk. you will not be able to use that port as a backup (your VSWITCH will not be able to set PRIROUTER with the other connection active on the port). we had two OSA Express ports at devices 2320-2322 and 2180-2182. and this conflicts with or does not match the configuration you need for the VSWITCH connection. or on other VSWITCHes. and they must all be configured to trunk the correct VLANs to the VSWITCH. the switch ports you connect your OSA Express adapters to must all be defined as trunk ports. Additional ports defined for the VSWITCH are not connected until required. to make your VSWITCH connectivity redundant.provided. Important: Only one port is used for communication between the VSWITCH and the LAN at any one time. to act as controllers.

The IUCV *VSWITCH directory entry is required to allow them to connect to the system service that manages VSWITCH connectivity to OSA Express adapters.STACK Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. You can add the definitions to the SYSTEM DTCPARMS file. as shown in Example 5. or use a machine-local DTCPARMS file containing the definitions on a local minidisk to the guest. the link command for the machines’ 198 disk must specify TCPMAINT 298 instead of TCPMAINT 198. of course) and update each guest’s directory entry to link to the correct minidisk.TCPCTL1 :TYPE. allowing the OSA Express to be defined to the TCP/IP machine with the same logical device numbers as the real OSA device addresses (RDEVs). you will need to have a separate 198 disk for each one. apart from the VSWITCH CONTROLLER ON statement required to allow the TCP/IP stack to initialize as a VSWITCH controller. This may be particularly relevant if you are using your production TCP/IP service machine as a VSWITCH controller. and it needs to be different on each of your controller service machines. Also. SYSTEM DTCPARMS Every TCP/IP service machine in your z/VM system needs to have DTCPARMS configured. you may need to do so to prevent the *VSWITCH system service from using device numbers that are already in use on your controller service machines.SERVER :DISKWARN.4 23 .NO :CLASS. Our DTCPARMS file was very simple. we used the default operation. We chose the latter method. This configuration allowed us to have an identical PROFILE TCPIP on all the controller service machines and share the configuration disk between all the controller service machines we set up. We used the following commands: FORMAT 298 K COPYFILE * * D = = K (OLDDATE # answer the prompts for volume label and confirm Directory entries We added the controller service machines to the z/VM directory as copies of the main TCPIP userid. as follows: LINK TCPMAINT 0298 0198 RR PROFILE TCPIP The PROFILE TCPIP for our controller service machines requires no special customization. Example 5 TCPCTL1 DTCPARMS file :NICK. creating TCPCTL1 DTCPARMS and TCPCTL2 DTCPARMS files on the controller service machines’ 198-disk. Attention: While we did not need to specify the LDEV range in the VSWITCH CONTROLLER parameter in our configuration. Important: If you specify the LDEV parameter.Setting up the configuration disk for the new service machines involved formatting the new minidisk and copying all the files from the configuration disk for the main TCP/IP service machine. Allocate additional TCPMAINT minidisks as copies of the TCPMAINT 198 disk as described in “Minidisk configuration” on page 22 (using different device numbers for each. We did not see a need in our configuration to restrict the machines to a range of LDEV (logical device) values.

Example 6 CP QUERY CONTROLLER q controller CONTROLLER TCPCTL1 Available: YES SYSTEM TESTSW3 Controller: * CONTROLLER TCPCTL2 Available: YES Ready. you will need to ensure that your switch configuration reflects the connectivity you want. Issue the CP QUERY CONTROLLER command to see that your service machines have initialized and that they have registered as controllers. Remember: PORTNAME must be specified last. Example 7 DEFINE VSWITCH results define vswitch testsw3 rdev 2320 2180 portname osa2320 osa2180 VSWITCH SYSTEM TESTSW3 is created Ready. we specified multiple values on the RDEV and PORTNAME parameters to provide access to our two OSA ports. Example 8 QUERY VSWITCH results query vswitch testsw3 VSWITCH SYSTEM TESTSW3 Type: VSWITCH PERSISTENT RESTRICTED NONROUTER State: Ready CONTROLLER: TCPCTL1 IPTIMEOUT: PORTNAME: OSA2320 RDEV: 2320 PORTNAME: OSA2180 RDEV: 2180 Ready. Some of the things you need to look at or check are described here. Example 7 shows the results of our DEFINE VSWITCH command. To get information about the z/VM Virtual Switch we just created.01 20:17:43 Active: 3 MFS: 8192 5 VDEV: 2320 MAXCONN: INFINITE ACCOUNTING: OFF QUEUESTORAGE: 8 Switch configuration To make sure you get the connectivity you expect in your environment. T=0.01 17:12:16 VDEV Range: * VDEV Range: * Configuring multiple OSA Express ports Remember. 24 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.01/0.01/0. T=0. When we defined our z/VM Virtual Switch using the CP DEFINE VSWITCH command. We will illustrate the settings you need to investigate with configuration statements we used in our switch.01 20:13:02 HCPSWU2830I VSWITCH SYSTEM TESTSW3 status is ready. Example 8 shows this output.Start the controller service machines Once configured.01/0.4 . last on DEFINE or SET VSWITCH commands. Example 6 shows the results we obtained after our controller service machines initialized. Important: Remember to add your new service machines to your automation procedures to ensure that they start automatically after an IPL of z/VM. the new controller service machines can be logged on using the XAUTOLOG command. HCPSWU2830I TCPCTL1 is VSWITCH controller. T=0. Note: Our testing environment used a Cisco Catalyst 6509 switch. we can issue the CP QUERY VSWITCH command.

1Q tagging.---------. VLAN numbers 1 through 1005 and 1025 through 4094 will be trunked on this port (this is the Cisco default).---------.---------.-----------------------202 active 114 2/3. however.-----.-------- Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.-------. Defining a VLAN Your LAN switch must include a definition for the VLAN you want to create. In Cisco CatOS. Example 9 Displaying VLAN status on a Cisco Catalyst switch itso6509 (enable) show vlan 202 VLAN Name Status IfIndex Mod/Ports.-----.----. You can use the show vlan command to display attributes of your VLAN configuration. The following command defined VLAN 202 on our switch.1025-4094 This command sets port 7 in module 2 to be a trunk port.---. Important: When using VLANs in z/VM Guest LANs. On our Cisco switch. using IEEE 802. Example 9 gives an example of this.--------. you must grant access to VLAN IDs as required.Setting up a trunk port If you are using VLANs within your VSWITCH. We specified an MTU of 1492 bytes to correspond with the MTU used on the OSA Express and VSWITCH adapters. Vlans ---. If you are using VLAN filtering. VLANs are defined using the set vlan command. we defined a trunk port with this command: set trunk 2/7 on dot1q 1-1005.4 25 . you do not need to pre-define the VLANs in your z/VM configuration.-----.-----. you will need to configure the switch port your OSA Express attaches to as a trunk port.2/7. some messages may appear when you enter a configuration command.2/11 15/1 VLAN Type SAID MTU Parent RingNo BrdgNo Stp BrdgMode Trans1 Trans2 ---.----. This would be useful to minimize unnecessary packet flow into the VSWITCH if you have many VLANs defined in the switch network but only some of them are represented in the VSWITCH. and you want to link the VLANs inside z/VM with VLANs in the LAN. This ensures two things: Frames sent to the OSA Express are VLAN tagged Frames for your desired VLANs are sent to the OSA Express You can configure the trunk port to carry a specific range of VLANs. itso6509 (enable) set vlan 202 type ethernet mtu 1492 Note: Depending on your logging level.------.-----202 enet 100202 1492 0 0 VLAN MISTP-Inst DynCreated RSPAN ---.-------------------------------.

202 - static disabled itso6509 (enable) There are many other configuration settings for VLANs available.-------.-------. This is shown by the <.4 ..------------.> Port Security Violation Shutdown-Time Age-Time Max-Addr Trap IfIndex ----. We used this routing function to provide access between our VLANs. or it may be a software or hardware component of the switch.------2/3 disabled shutdown 0 0 1 disabled 12 <. Routing to and from your VLAN Stations attached to different VLANs cannot communicate to each other without a router. Example 10 gives a sample of this display..> symbols in the output... 06:56:44 Idle Detection --------------itso6509 (enable) We have truncated some of the output of this command. Example 10 Displaying port information on a Cisco Catalyst switch itso6509 (enable) show port 2/3 * = Configured MAC Address Port Name Status Vlan Duplex Speed Type ----.---------.-------.-------------------. itso6509 (enable) set vlan 202 2/3 The show port commands will display various information about your port status..-----.> Port Last-Time-Cleared ----.-----------2/3 connected 202 a-full a-100 10/100BaseTX <.---------. Refer to your Cisco documentation for more detail.--------.-------.-------------------------2/3 Sun Jul 6 2003. which provides a routing component. We used the CatOS set vlan command to assign switch ports to VLANs. The router may be a physical device with attachments to ports on the switch. The following command will attach port 3 on module 2 to VLAN 202. 26 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.. The Cisco Catalyst switch in our lab was equipped with the Multilayer Switch Feature Card (MSFC).----. Adding ports to a VLAN One of the first things you might want to do is configure certain ports on your switch to access certain VLANs.

The diagram in Figure 6 on page 28 shows our network testing configuration. Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. 0 packets/sec 556993 packets input. 0 output buffers swapped out itsores1> There are many other commands you can use to get information about your routing configuration. 38350 bytes.202. one per line. BW 1000000 Kbit. Example 11 Configuring MSFC for VLAN routing itsores1#configure terminal Enter configuration commands. Example 12 shows a sample of this. Example 12 Interface status for a MSFC VLAN virtual interface itsores1>show interface vlan202 Vlan202 is up.4 27 . The interface vlan command is used to create a VLAN virtual interface.0 itsores1(config-if)#^Z itsores1# Additional commands are required to enable routing in the MSFC. Refer to your Cisco documentation for more information.255.168. 0 frame. 0 overrun.168.7302) Internet address is 192. End with CNTL/Z.85f3. DLY 10 usec.1 255. 0 runts. 2 interface resets 0 output buffer failures. reliability 255/255. 0 no buffer Received 556777 broadcasts. 28982480 bytes. You can display the status of your VLAN virtual interface using the show interface command. txload 1/255. rxload 1/255 Encapsulation ARPA. and to set up a dynamic routing protocol if you use one. output never.7302 (bia 0007.Configuring a connection to a VLAN for routing requires adding a virtual interface to the MSFC representing a connection to the VLAN. ARP Timeout 04:00:00 Last input 00:00:01. line protocol is up Hardware is Cat6k RP Virtual Ethernet. Example 11 shows the commands we used to add a VLAN interface to the configuration of the routing function in our switch. address is 0007. 0 CRC. Total output drops: 0 Queueing strategy: fifo Output queue :0/40 (size/max) 5 minute input rate 0 bits/sec.85f3.255. 0 ignored 465 packets output. output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes).1/24 MTU 1500 bytes. and the ip address command is used to assign an IP address to the virtual interface. Refer to Cisco documentation if you need assistance with these. 0 underruns 0 output errors. itsores1(config)#interface vlan 202 itsores1(config-if)#ip address 192. Experiences This section describes some aspects of VSWITCH and HiperSockets VLAN support that we encountered while writing this paper. 0 packets/sec 5 minute output rate 0 bits/sec. 0 throttles 0 input errors.202. 0 giants. loopback not set ARP type: ARPA.

35 LNXSU3 LNXSU2 HiperSockets Guest LAN ITSO — 1 2 201 202 LNXSU1 TCPIP VSWITCH TESTSW3 Linux Guests 192.9. We made sure that the LAN PC was only reachable via the VLAN link via the VSWITCH (that is.200.168.0/24 Switch/Router Routing Cisco Catalyst 6509 Windows Test Station 192. but because we granted it access to VLAN ANY it can also generate and receive untagged frames.168.46 .202.12.37 LNXSU4 LNXRH2 . 28 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. the ability to generate and receive untagged frames is shown by membership to the ‘—’ VLAN. The diagram also shows a HiperSockets Guest LAN with a single VLAN.168.4 .9. the guest LNXSU7 belongs to VLAN 201. as well as the relationship between the VSWITCH OSA Express port and the controller TCPIP service machine.0/24 Figure 6 Test network configuration In this diagram. we represented the VLANs in a switch (including z/VM Virtual Switch) as labelled rectangles within the representation of the switch.100. For example. A particular station or guest’s membership of a VLAN is indicated by a solid circle in the rectangle for the VLAN intersecting the connection from the station. was granted access to VLAN 201 only.45 LNXSU7 . on the other hand.44 LNXSU6 . LNXRH2. We used the ping command from one of the Linux guest machines on our VSWITCH to a PC attached to a port on the LAN switch.0/24 VMLINUX HiperSockets 0 2211 . z/VM Virtual Switch failover With the configuration as described in “Configuring high availability with VSWITCH” on page 22.36 .0/24 OSA VLANs OSA 1 2 201 202 Linux Test Servers 192.38 . that there was no alternative path for the connection to travel over). we tried various ways to disrupt the communication through the OSA Express port to the LAN. and verified this by using the -R switch on the ping command. In the VSWITCH.

Tip: Adding the -R switch on the ping command adds the RECORD_ROUTE option to the ICMP echo request packets that ping generates.168.01/0.100: 192. it can display the route taken by the packet.168.100: 192.4 29 .200.200.168.100 ping statistics --13 packets transmitted.168. time 12006ms rtt min/avg/max/mdev = 0. Deactivate the switch port in the LAN switch This failure is detected as a loss of signal or cable pull on the OSA Express.631 ms icmp_seq=12 ttl=64 time=0. if your round trip exceeds nine hops.168. and sometimes the return path is different).608 ms icmp_seq=3 ttl=64 time=0.168.100) from 192.592 ms icmp_seq=4 ttl=64 time=0.200.14 ms RR: 192. the connection was transferred to the other OSA.592/0. q vswitch testsw3 <. The effect is similar to traceroute.168.100: 192.200. 64 bytes from 192.168.100: 192.597 ms icmp_seq=11 ttl=64 time=0.100: icmp_seq=2 ttl=64 time=0.100: 192.109 192.100: 192. Example 13 Ping test across OSA Express port detach lnxsu6:~ # ping -R 192.168.168.168.200. Unfortunately.100 PING 192.623 ms (same (same (same (same (same (same (same route) route) route) route) < disable here route) route) route) --. 8 received.192.200. only nine hops can be counted using ping -R.624 ms icmp_seq=13 ttl=64 time=0.200.disable here \ unsolicited / messages / Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. On a z/VM console.200.168.168.200. We used the set port disable command in the Cisco Catalyst switch to shut down the switch port.200.168.before disable MAXCONN: INFINITE ACCOUNTING: OFF QUEUESTORAGE: 8 attached.613 ms icmp_seq=5 ttl=64 time=0. so that when the reply is received by ping. 38% loss.200. and the VSWITCH swaps to the other OSA port.147/0.200.200. Example 14 z/VM commands and messages around switch port deactivation q vswitch testsw3 VSWITCH SYSTEM TESTSW3 Type: VSWITCH Active: 1 PERSISTENT RESTRICTED NONROUTER MFS: 8192 State: Ready CONTROLLER: TCPCTL1 IPTIMEOUT: 5 PORTNAME: OSA2180 RDEV: 2180 PORTNAME: OSA2320 RDEV: 2320 VDEV: 2320 Ready. you will only see the first nine. T=0.168.168.200.100 192. \ <. Example 13 shows the results or our ping test. Each host along the path adds its address to the IP header.100 (192. but the response is much quicker and you obtain round-trip data (traceroute will only tell you the path from source to destination. HCPSWU2830I TCPCTL1 is VSWITCH controller.100 192.109 : 56(124) bytes of data.200. we issued the commands and received messages as shown in Example 14.01 15:45:31 HCPSWU2830I VSWITCH SYSTEM TESTSW3 status is devices HCPSWU2830I TCPCTL1 is VSWITCH controller.200.100: icmp_seq=1 ttl=64 time=1.109 64 64 64 64 64 64 64 bytes bytes bytes bytes bytes bytes bytes from from from from from from from 192.200.679/1.178 ms lnxsu6:~ # You can see that after a short period of time. HCPSWU2830I VSWITCH SYSTEM TESTSW3 status is ready.168.

168. LAN communications are re-established. Status: Not started DTCPRI387I Envelope queue size: 0 DTCPRI388I Address: 2180 DTCQDI001I QDIO device TESTSW32180DEV device number 2182: 6 DTCQDI007I Enabled for QDIO data transfers DTCOSD238I ToOsd: Multicast support enabled for TESTSW32180DEV DTCOSD319I ProcessSetArpCache: Supported for device TESTSW32180DEV DTCOSD341I Obtained MAC address 0006296CA5BC for device TESTSW32180DEV DTCOSD246I VSWITCH-OSD device TESTSW32180DEV: Assigned IPv4 address 192. This output is shown in Example 15. The IP addresses for active guests are registered to the OSA Express port.168.VSWITCH SYSTEM TESTSW3 Type: VSWITCH PERSISTENT RESTRICTED NONROUTER State: Ready CONTROLLER: TCPCTL1 IPTIMEOUT: PORTNAME: OSA2180 RDEV: 2180 PORTNAME: OSA2320 RDEV: 2320 Ready. we saw some very good message output. TCP/IP commences a shutdown of the VSWITCH dynamically-defined device. QDIO data transfer on the failed interface is terminated. 4. 3. T=0.01 15:48:11 Active: 1 MFS: 8192 5 VDEV: 2180 MAXCONN: INFINITE ACCOUNTING: OFF QUEUESTORAGE: 8 In the console log of the TCPIP service machine. TCP/IP initializes a new dynamically-defined device for the newly-attached OSA Express port. Detach the OSA from the controller service machine The next thing we tried was to detach the real devices for the VSWITCH’s active OSA Express port from the system.4 .202. Status: Ready DTCPRI387I Envelope queue size: 0 DTCPRI388I Address: 2320 DTCQDI001I QDIO device TESTSW32320DEV device number 2322: 3 DTCQDI007I Disable for QDIO data transfers DETACHED TCPCTL1 2320 BY TCPCTL1 4 DETACHED TCPCTL1 2321 BY TCPCTL1 DETACHED TCPCTL1 2322 BY TCPCTL1 DTCOSD080I VSWITCH-OSD initializing: 5 DTCPRI385I Device TESTSW32180DEV: DTCPRI386I Type: VSWITCH-OSD. 30 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. QDIO data transfer for the new OSA Express port is activated. Example 15 TCPIP console log output across switch port deactivation 15:47:52 15:47:52 15:47:52 15:47:52 15:47:52 15:47:52 15:47:52 15:47:52 OSA 2320 OSA 2321 OSA 2322 15:47:52 15:47:52 15:47:52 15:47:52 15:47:52 15:47:52 15:47:52 15:47:52 15:47:52 15:47:52 15:47:57 15:47:57 15:47:57 DTCOSD309W Received adapter-initiated Stop Lan 1 DTCOSD082E VSWITCH-OSD shutting down: 2 DTCPRI385I Device TESTSW32320DEV: DTCPRI386I Type: VSWITCH-OSD. 2.109 DTCOSD246I VSWITCH-OSD device TESTSW32180DEV: Assigned IPv4 address 192. 6. 7.109 7 The chain of events is as follows: 1.109 DTCOSD246I VSWITCH-OSD device TESTSW32180DEV: Assigned IPv4 address 192.01/0. This allowed us to simulate the OSA Express device becoming unavailable through hardware failure or operator intervention.201.168. The OSA Express notifies TCP/IP that the LAN is no longer available. TCP/IP detaches the real OSA Express devices and attaches the devices for the backup OSA Express port (only the device detachments were logged). 5.200.

100: 192.621 ms (same (same (same (same (same route) route) <.02 ms RR: 192. The result of the ping test is shown at Example 16.100: 192.168.100 ping statistics --11 packets transmitted.100: 192.200.100: icmp_seq=2 ttl=64 time=0.200.625 ms icmp_seq=3 ttl=64 time=0.100 (192. time 10006ms rtt min/avg/max/mdev = 0. and communications would fail). T=0.168.100 192.200.200. We found that repeatedly detaching the OSA Express devices from the TCPIP service machines eventually would cause problems with the OSA (channel errors were reported in the TCPIP log when the OSA was used.705/1. Almost as soon as the disruption took place.109 192. we issued the commands and received messages as shown in Example 17.force here route) route) route) --.100 192. take care with your OSA connections. Again we used ping to verify the communication path.146 ms lnxsu6:~ # You can see that communication was interrupted for a short time while another controller TCPIP stack took control of the VSWITCH.200. 64 bytes from 192. Failure of the controller service machine We simulated the failure of the controller service machine by using the CP FORCE command on its userid.200.109 : 56(124) bytes of data. 6 received.01 15:48:11 HCPSWU2830I VSWITCH SYSTEM TESTSW3 status is controller not available.200.168. we had to reset the OSA by configuring the channel path offline and back online.168. Example 17 z/VM commands and messages around controller failure q vswitch testsw3 VSWITCH SYSTEM TESTSW3 Type: VSWITCH Active: 1 MAXCONN: INFINITE PERSISTENT RESTRICTED NONROUTER MFS: 8192 ACCOUNTING: OFF State: Ready CONTROLLER: TCPCTL1 IPTIMEOUT: 5 QUEUESTORAGE: 8 PORTNAME: OSA2180 RDEV: 2180 VDEV: 2180 PORTNAME: OSA2320 RDEV: 2320 Ready.200.100 PING 192.192. \ <.025/0.200.168.168.168.The results were the same as for the switch port deactivation.168.100) from 192.697 ms icmp_seq=10 ttl=64 time=0. 45% loss. On a z/VM console.200. Example 16 Ping test across controller service machine outage lnxsu6:~ # ping -R 192. > unsolicited HCPSWU2830I TCPCTL2 is VSWITCH controller.621/0.168.168.100: icmp_seq=1 ttl=64 time=1.623 ms icmp_seq=11 ttl=64 time=0.01/0.168.force here HCPSWU2830I VSWITCH SYSTEM TESTSW3 status is ready. the OSA Express detected a loss of communication and the VSWITCH was transferred to the other OSA Express port.200.168. To recover.4 31 . / messages q vswitch testsw3 VSWITCH SYSTEM TESTSW3 Type: VSWITCH Active: 1 MAXCONN: INFINITE PERSISTENT RESTRICTED NONROUTER MFS: 8192 ACCOUNTING: OFF State: Ready CONTROLLER: TCPCTL2 IPTIMEOUT: 5 QUEUESTORAGE: 8 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. Attention: If you want to conduct similar testing.200.168.641 ms icmp_seq=9 ttl=64 time=0.200.168.200.168.200.100: 192.109 64 64 64 64 64 bytes bytes bytes bytes bytes from from from from from 192.

there were two other guests on the VLAN.12. We configured the Red Hat guest with an IP address configuration for that VLAN.NOARP.UP> mtu 1492 qdisc pfifo_fast qlen 100 link/ether 02:00:00:00:00:10 brd ff:ff:ff:ff:ff:ff inet 192.NOARP.168.10 9.168.168. To test this.0/24 dev hsi0 scope link 192.0/24 via 192.201.255 scope host lo 2: eth4: <MULTICAST. especially with non-VLAN-aware guests.168.1/8 brd 127.168.0. Before you rely on this function to protect critical services. As distributed.168.NOARP> mtu 1492 qdisc noop qlen 100 link/ether 02:00:00:00:00:0f brd ff:ff:ff:ff:ff:ff 4: hsi1: <MULTICAST.202.9.100.202.12. both VLAN-aware SLES8 systems.168.255.0/24 dev eth4 proto kernel scope link src 192.0.9.0/24 dev eth4 scope link 192.UP> mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.200. Example 18 IP configuration on LNXRH2 [root@lnxrh2 root]# ip addr ls 1: lo: <LOOPBACK. Ping is only a very simplistic test. We then tried configuring different IP addresses as secondary addresses on the eth4 interface of the guest (the interface that connected to the VSWITCH).45/24 brd 9. Red Hat 7.10 127.4 . Other than our Red Hat guest.0 dev eth4 192. When we granted access to the VSWITCH.9.255 scope global hsi0 [root@lnxrh2 root]# ip route ls 192.2 has no support for VLANs. We wanted to verify what happened when untagged frames were used. this was the guest LNXRH2). conduct your own testing to verify that the protocols and applications that you run in your environment recover from the brief outage interval. T=0.2 to our VSWITCH (in Figure 6 on page 28. and verified that we could communicate between it and other guests on the VLAN.168.12.10/24 scope global eth4 inet 192. VLAN Isolation with untagged frames When setting up and using VLANs in our testing with VSWITCH.01 16:47:38 Important: We are very impressed with the way that z/VM Virtual Switch provides failover for its components.9.0.0/24 dev eth4 proto kernel scope link src 192.0/8 dev lo scope link default via 9.10/24 scope global eth4 3: eth2: <MULTICAST. we attached a guest running Red Hat Linux 7.202.10/24 brd 192.201.168.12.100. the connectivity between the VSWITCH and the LAN is restored.1 dev hsi0 [root@lnxrh2 root]# 32 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.201.100.255.201.NOARP> mtu 8192 qdisc noop qlen 100 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff 5: hsi0: <MULTICAST. we were very satisfied with how the system preserved isolation of the different VLANs used.PORTNAME: OSA2180 RDEV: 2180 VDEV: 2180 PORTNAME: OSA2320 RDEV: 2320 Ready.UP> mtu 16384 qdisc pfifo_fast qlen 100 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff inet 9.255 scope global eth4 inet 192.168.01/0.168.0. with only a short interval of dead time. we granted it only to VLAN 201.

10 Mask: 255.0/24 network.109) from 192.100. You can inspect this table using the CP QUERY VSWITCH command with the DETAIL option. This confirmed that we were being isolated from VLAN 202. Example 20 CP QUERY VSWITCH for IP address table q vswitch testsw3 detail VSWITCH SYSTEM TESTSW3 Type: VSWITCH Active: 3 MAXCONN: INFINITE PERSISTENT RESTRICTED NONROUTER MFS: 8192 ACCOUNTING: OFF State: Ready CONTROLLER: TCPCTL1 IPTIMEOUT: 5 QUEUESTORAGE: 8 PORTNAME: OSA2320 RDEV: 2320 VDEV: 2320 PORTNAME: OSA2180 RDEV: 2180 RX Packets: 30569 Discarded: 33 Errors: 0 TX Packets: 27494 Discarded: 0 Errors: 0 RX Bytes: 2744464 TX Bytes: 2305386 Authorized userids: LNXRH2 VLAN: 0201 LNXSU6 VLAN: ANY LNXSU7 VLAN: 0201 0202 SYSTEM VLAN: ANY VSWITCH Connection: Device: 2322 Unit: 002 Role: DATA VLAN: ANY Assigned by user Unicast IP Addresses: 192.168.168.109: icmp_seq=2 ttl=64 time=275 usec --.255.041 ms [root@lnxrh2 root]# To try and understand what was happening here.0.0 192. even though we had restricted the guest to VLAN 201.255.100.100.100.4 33 .0/24 network.168.109: icmp_seq=1 ttl=64 time=292 usec 64 bytes from 192.168.0 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.202 Mask: 0.168.10 Mask: 255. Example 19 Ping command on LNXRH2 [root@lnxrh2 root]# ping 192.168.359/0.0 192.109: icmp_seq=0 ttl=64 time=359 usec 64 bytes from 192.168.202.255. we were surprised to find that we did get responses.Our Red Hat guest was authorized only to VLAN 201.168.0.168.10 : 56(84) bytes of data.255. The VSWITCH maintains a table of connected guests and associated IP addresses. By configuring IP addresses belonging to other VLANs against our interface.201.168.202. Example 20 shows this command output after the ping command above.202. corresponding to the 192.201.168.100.10 Mask: 255.100. The VSWITCH uses this table to determine which guest to send a received packet to.255.0 Remote Adapter Owner: LNXRH2 NIC: 6100 Name: TESTSW3 Device: 6102 Unit: 002 Role: DATA VLAN: 0201 Assigned by system Unicast IP Addresses: 192. we got no responses. However.168.0/24 network.100.308/0.109 PING 192.168. When we tried to ping the hosts on the 192.100. 0% packet loss round-trip min/avg/max/mdev = 0.275/0.100. 64 bytes from 192.109 (192.168. 3 packets received. We should only be able to communicate directly with other guests on that same VLAN.100.192.255. when we then tried to ping a host on the 192.109 ping statistics --3 packets transmitted. we had to look closer into the action of both the VSWITCH code and the Linux kernel. we tell the kernel to try and reach those networks directly instead of routing.168.

01/0.1 MAC: 01-00-5E-00-00-01 Adapter Owner: LNXSU6 NIC: 6100 Name: TESTSW3 Device: 6102 Unit: 002 Role: DATA VLAN: ANY Assigned by user Unicast IP Addresses: 192. The packet is given to the VSWITCH.109 Mask: 255.0. the kernel will simply accept it at the interface it was delivered.0.201. it technically has an incorrect destination address for the interface it is arriving on. Untagged frames are not filtered by VSWITCH. so the packet is given to its destination (the ping program).168. which it is in this case.100. The kernel processes the ICMP echo request.255. since the two machines were configured with a common network interface and were able to communicate anyway. so VSWITCH delivers the frame.168.0 FE80::200:0:300:8/0 Local Multicast IP Addresses: 224. The IP stack in LNXRH2 checks to see if the destination IP address is local. The address is local. formatting an ICMP echo reply to be sent to 192.168. It checks the IP address table again. the fact this happens this way is a curiosity.168.168. so the frame is sent to LNXRH2. 5. because we have that address configured on one of our local interfaces and the route table selected it. VSWITCH takes no action on the untagged frame entering the VLAN. This packet will be sent as an untagged frame.100.255. Because LNXSU6 is authorized to VLAN ANY. so the packet handled by the kernel.0.255.168. the VSWITCH looks up the IP address table for the destination address.0. so the next check is whether the VLAN authority of LNXSU6 will allow the frame to be delivered. However if the machines were not intended to have a shared network then this could lead to unexpected connectivity and a loss of security.202. 3. 1. In this instance. LNXSU6 is authorized to VLAN ANY. 34 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.10.168. and sends the reply via the eth4 interface.109 as the source address. 2. At step 3. 6.4 .100. when the packet arrives at the VLAN interface.1 MAC: 01-00-5E-00-00-01 FF02::1 MAC: 33-33-00-00-00-01 Local FF02::1:FF00:8 MAC: 33-33-FF-00-00-08 Local Ready.109 Mask: 255. and finds it on LNXRH2. Rather than forwarding the packet Many IP stacks through the stack to deliver it at the correct interface. 4.100. The source address is 192.0 192. The route table selects 192.109 Mask: 255.10. this time for the destination address 192.Multicast IP Addresses: 224.01 15:22:41 The following chain of events takes place during our ping.255.100.10. Next. LNXRH2 sends an ICMP echo request packet.255. The main reason this connection can take place is an efficiency aspect of the Linux kernel. T=0. The IP stack in LNXSU6 does a final check to see if the destination address is local or if the packet has to be routed. work this way to improve performance. LNXSU6 has registered this address.0 192.255. The other reason that the connection works is that the VSWITCH does not filter untagged frames for delivery to guests. and because LNXRH2 is authorized to only one VLAN ID the VSWITCH tags the frame for VLAN 201.

Attention: To use tcpdump with QETH network interfaces. this configuration eliminated untagged frames from LNXSU6.Closing the door There are at least two ways you can eliminate this possible exposure: by eliminating untagged frames. The frames sent back to LNXRH2 were not delivered by the VSWITCH because they were tagged as VLAN 1. we could still communicate with the other machines on the 192. we do not believe that a design problem exists. By changing the configuration of LNXSU6 to bring up a connection to the 192. Rather. Eliminate untagged frames In the configuration of LNXSU6. and the Ethereal analysis tool to look at the trace. In the following sections.100. as far as the switch was concerned. Importantly. we are drawing your attention to an interoperation aspect that you should be aware of when implementing VSWITCH and VLAN in a Linux environment.100. This script is included in the tcpdump package on SuSE SLES 8.168. Figure 7 on page 36 is a screen capture of the Ethereal program analyzing our captured traffic. Analyzing the captured packets.0/24 network). Capturing VLAN network traffic on VSWITCH We used the tcpdump packet capture tool to grab network traffic. you can prevent the kernel from accepting packets on an interface whose network does not match the destination address of the packet. a VLAN that LNXRH2 was not authorized to.4 35 .0/24 network.168.0/24 network). The Cisco switch that we connected to was defined to use VLAN 1 as a default VLAN ID. Firewall rules in Linux Many administrators do not like the default behavior of Linux accepting packets on “foreign” interfaces. Attention: We are not trying to point out a design problem by illustrating this issue.168. and issued ping commands for destinations on the various VLANs that our host was attached to.0/24 network via VLAN 1 and using a dummy IP address on the base interface.200. the base interface (eth4) was used to gain connectivity to the other Linux guests that were attached to the other OSA device in the network (the 192. so all the Linux guests attached to the other OSA were actually in VLAN 1. By adding iptables rules in any Linux host attached to multiple VLANs.100. or by using firewall rules in Linux.168. We started a packet capture against the VLAN 2 interface (the 192. the tcpdump-qeth wrapper script by Holger Smolinski of IBM is needed to analyze the frames correctly. we discuss these methods in more detail. we only saw the packets on VLAN2. This was what we expected. Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.

255. Looking at the statistics for the VLAN 2 interface. This is the dummy Ethernet header inserted into the frame by the tcpdump-qeth script.0 inet6 addr: fe80::200:0:300:8/10 Scope:Link 36 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. which is Linux receiving packets on any configured interface as long as the IP address is local. you can see that the Ethernet II header has all-zero MAC addresses for source and destination MAC. Note: In the Ethereal display shown in Figure 7. it would seem that the frames are being delivered to the base interface and accepted there rather than registering against the VLAN interface.200. this definitely appears to be the case (refer to Example 21). In this case. You can also see that we only appear to have captured the transmitted packets. is apparent here.255.168.Figure 7 Ethereal packet analysis on VLAN 2 We highlighted one packet for display in the decode window. It appears that the same “feature” that we discussed in “VLAN Isolation with untagged frames” on page 32.4 . Example 21 Interface statistics for a VLAN interface lnxsu6:~ # ifconfig vlan2 vlan2 Link encap:Ethernet HWaddr 02:00:00:00:00:08 inet addr:192.109 Mask:255.

Seeing this behavior. In doing so we hoped to see frames with VLAN headers in the trace.8 Kb) You can see that no received packets have been recorded against this interface. we ran tcpdump with the -p option.0 b) TX bytes:664403 (648. 2 Promiscuous mode gave the same result.4 37 . which shows it is fairly meaningless on z/VM simulated networks anyway.UP RUNNING MULTICAST MTU:1492 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:7783 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0. we then traced the base interface. Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. In particular. which prevents tcpdump from using promiscuous mode (we wanted to be absolutely sure we were not opening anything up that we did not intend to2). We wanted to see the contents of the VLAN headers and other changes to packets that the VLAN support made. Two things surprised us about what we saw in the trace of the base interface: We saw no VLAN headers We saw both transmitted and received IP packets for all VLANs Figure 8 on page 38 shows Ethereal analyzing a packet trace we collected against the eth4 interface on LNXSU6.

this makes sense. having extracted the VLAN headers and leaving just the IP payload. the QETH driver has been released as open source. but of other frame headers as well. By the time tcpdump views the packets. Note: With the IBM s390 kernel patches for the 2. not all frames received on a QETH interface will ever have had an Ethernet header: traffic between guests on the same simulated network.21 kernel.4 . Also. Since QETH is an IP-only transport. the 802. It appears that tcpdump can only see IP frames when QETH is used. However. This may allow better visibility not only of VLAN frames. this still may not produce the desired results because the Ethernet header is not always preserved by the OSA Express hardware. An interesting project would be to modify the driver so that “traditional” tools like libpcap function as expected. do not. 38 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. you can see some of the packets that were collected.4.Figure 8 Ethereal packet analysis on base interface In the top portion of the window. for example.1Q processing has finished with them. The source and destination addresses for these packets are for addresses on VLAN 2 and VLAN 202 in our configuration.

Example 23 SET VSWITCH syntax SET VSWITCH . .-GRAnt--userid--+------------------+-. | | | '-VLAN----vlanid-+-' | |-REVoke--userid----------------------| | .--------------.---rdev-----+. Refer to z/VM CP Command and Utility Reference for full details of all parameters.Command usage The usage syntax of the commands used in this paper are shown in this section.-NONrouter-.----------------------------->< '-PRIrouter-' | .-RDEV--NONE--------.-------------. . | | V (3)| | '-PORTname----portname----+-' Notes: (1) You can specify the operands in any order.-CONnect----. and portname is the last operand specified. You can specify a maximum of 3 port names. | '-DISCONnect-' | V (2)| | '-RDEV----rdev----+-' . Example 22 DEFINE VSWITCH syntax DEFINE VSWITCH (1) .-CONTRoller--*-------.-VLAN--ANY--------. >>--SET VSWITCH--switchname--. >>--DEFINE VSWITCH--switchname------+-------------------+--+------------+----> | . >--+-----------+--.4 39 . | | V (2) | | |-RDEV--.-IPTimeout--5---.---------------| | '-NONE--------' | Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.-------->< | | v--------.-QUEuestorage--8M------. DEFINE VSWITCH The syntax of the DEFINE VSWITCH command is shown in Example 22.----------.---------------------------. SET VSWITCH The syntax of the SET VSWITCH command is shown in Example 23.---------. >--+-----------------------+--+---------------------+--+----------------+----> '-QUEuestorage--numberM-' '-CONTRoller--userid1-' '-IPTimeout--nnn-' . if applicable. as long as switchname is the first operand specified. | | V (1) | | |-PORTname----portname-----+----------| | . (2) (3) You can specify a maximum of 3 real device numbers. .

|---.-------------| | '-userid1-' | |-IPTimeout--nnn----------------------| |-NONrouter---------------------------| '-PRIrouter---------------------------' Notes: (1) You can specify a maximum of 3 port names. Usage: add rem Was: 1 [interface-name] [vlan_id] [vlan-name] 40 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.-' '-QDIO---------' '-| NICDEF Options |-' NICDEF Options: v----------------------------------.----------------------->< '-TYPE--.-DELETE-----------------------------------------.-| | | |-*-------| | | | | '-SYSTEM--' | | | '-SYSTEM--switchnm-----' | |-CHPID--hh---------------------| '-MACID--macaddress-------------' Note: (1) For more information on prefix keywords.|-CONnect-----------------------------| |-DISCONnect--------------------------| |-QUEuestorage--numberM---------------| |-CONTRoller--.-HIPERsockets-. Refer to z/VM CP Command and Utility Reference for full details of all parameters. enter the following command: dirm help dirmaint vconfig Example 25 shows the usage of the vconfig command. inclusive.-*-------.--------------------.4 .-+--------------------------------------| |-LAN--. (2) You can specify a maximum of 3 real device numbers.-DEVices--devs-----------------.-ownerid-.-.--NICDEF--vdev---------------------------> | (1)| '-Prefix Keywords----' >--. DIRMAINT NICDEF Example 24 shows the options available on the DIRMAINT NICDEF command.--lanname-.--.--------------------. Example 24 DIRMAINT NICDEF command NICDEF >>--DIRMaint--. Example 25 Usage of the vconfig command lnxsu6:~ # vconfig Expecting argc to be 3-5.

* vlan_qos is the 3 bit priority in the VLAN header * name-type: VLAN_PLUS_VID (vlan0005). * The vlan_id is the identifier (0-4095) of the VLAN you are operating on. If you don't need it. IBM’s Large Systems Business Partner in Australia. He has more than 15 years of experience in general computing. DEV_PLUS_VID_NO_PAD (eth0. He holds a Bachelor of Computing Science degree from Queensland University of Technology. IBM Endicott Simon Williams IBM Australia Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. Default is OFF. Poughkeepsie Center. Vic Cross is the Linux for zSeries and S/390® Team Leader at Independent Systems Integrators. The author of this Redpaper This Redpaper was produced by a team of specialists from around the world working at the International Technical Support Organization. Thanks to the following people for their contributions to this project: Roy Costa and Greg Geiselhart International Technical Support Organization. the VLAN device will move the ethernet header around to make it look exactly like a real ethernet device. SG24-6299. PER_KERNEL # Forces vlan 5 to be unique across all devices. because there will be at least a small performance degradation. * skb_priority is the priority in the socket buffer (sk_buff). and Linux on IBM ^ zSeries and S/390: Virtual Router Redundancy Protocol on VM Guest LANs. SG24-6824. Linux on IBM ^ zSeries and S/390: Large Scale Linux Deployment. Poughkeepsie Center Angelo Macchiano and Dennis Musselwhite z/VM Development. * FLAGS: 1 REORDER_HDR When this is set. He is a co-author of IBM Redbooks™ and Redpapers Linux on IBM ^ zSeries and S/390: ISP/ASP Solutions. eight of which have been spent working on S/390 and zSeries. REDP3657.set_flag set_egress_map set_ingress_map set_name_type [interface-name] [flag-num] [vlan-name] [skb_priority] [vlan-name] [skb_priority] [name-type] [0 | 1] [vlan_qos] [vlan_qos] * The [interface-name] is the name of the ethernet card that hosts the VLAN you are talking about.5) * bind-type: PER_DEVICE # Allows vlan 5 on eth0 and eth1 to be unique.4 41 . REDP3627. Linux on IBM ^ zSeries and S/390: Porting LEAF to Linux on zSeries. His areas of expertise include networking and Linux. DEV_PLUS_VID (eth0. don't turn it on. This may help programs such as DHCPd which read the raw ethernet packet and make assumptions about the location of bytes.0005). VLAN_PLUS_VID_NO_PAD (vlan5).

IBM Redbooks Linux on IBM ^ zSeries and S/390: Virtual Router Redundancy Protocol on VM Guest LANs.4 .Related publications The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this Redpaper. REDP3657 Other publications z/VM Virtual Machine Operation. SC24-6008 42 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. SC24-5955 z/VM CP Command and Utility Reference.

or service. Some states do not allow disclaimer of express or implied warranties in certain transactions.Notices This information was developed for products and services offered in the U. or service that does not infringe any IBM intellectual property right may be used instead. modify. it is the user's responsibility to evaluate and verify the operation of any non-IBM product. IBM Corporation. therefore. these changes will be incorporated in new editions of the publication. Any reference to an IBM product. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND. to: IBM Director of Licensing. EITHER EXPRESS OR IMPLIED. However. services. This information contains examples of data and reports used in daily business operations. THE IMPLIED WARRANTIES OF NON-INFRINGEMENT. Information concerning non-IBM products was obtained from the suppliers of those products. or service may be used. program. program. cannot guarantee or imply reliability. IBM has not tested those products and cannot confirm the accuracy of performance. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. or features discussed in this document in other countries. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. this statement may not apply to you. using. program.S. or function of these programs. COPYRIGHT LICENSE: This information contains sample application programs in source language. INCLUDING. and distribute these sample programs in any form without payment to IBM. 2003. serviceability. and distribute these sample programs in any form without payment to IBM for the purposes of developing. All rights reserved. marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. You may copy. Consult your local IBM representative for information on the products and services currently available in your area. NY 10504-1785 U. BUT NOT LIMITED TO. marketing. 43 .A. You may copy. IBM may not offer the products. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. using. You can send license inquiries. in writing. This information could include technical inaccuracies or typographical errors. To illustrate them as completely as possible. North Castle Drive Armonk. Changes are periodically made to the information herein. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. companies. program. © Copyright IBM Corp. for the purposes of developing. or service is not intended to state or imply that only that IBM product. IBM.S. Any functionally equivalent product. the examples include the names of individuals. Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. brands. MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.A. These examples have not been thoroughly tested under all conditions. IBM may have patents or pending patent applications covering subject matter described in this document. compatibility or any other claims related to non-IBM products. and products. which illustrates programming techniques on various operating platforms. modify. therefore. their published announcements or other publicly available sources. The furnishing of this document does not give you any license to these patents. or distributing application programs conforming to IBM's application programming interfaces.

other countries.com® IBM® Lotus® MVS™ Notes® Redbooks(logo) Redbooks™ S/390® ™ ™ z/OS® z/VM® zSeries® The following terms are trademarks of other companies: Intel. 44 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. product. other countries. and service names may be trademarks or service marks of others. NY 12601-5400 U. in the United States. MMX. Windows NT. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems. and the Windows logo are trademarks of Microsoft Corporation in the United States. 2003.A. International Technical Support Organization Dept.4 . SET Secure Electronic Transaction. and the SET Logo are trademarks owned by SET Secure Electronic Transaction LLC. Send us your comments in one of the following ways: Use the online Contact us review redbook form found at: ibm. UNIX is a registered trademark of The Open Group in the United States and other countries. Other company.ibm. HYJ Mail Station P099 2455 South Road Poughkeepsie. and Pentium are trademarks of Intel Corporation in the United States. Microsoft. or both. Windows. ® Trademarks The following terms are trademarks of the International Business Machines Corporation in the United States. or both.This document created or updated on September 3. SET.S. other countries. or both.com/redbooks Send your comments in an Internet note to: redbook@us. other countries. Inc.com Mail your comments to: IBM Corporation. or both: Domino™ HiperSockets™ ibm. Intel Inside (logos).