You are on page 1of 44

Redbooks Paper

Vic Cross

Linux on IBM

zSeries and S/390:

VSWITCH and VLAN Features of z/VM 4.4
Introduction and overview
z/VM® Virtual Switch (VSWITCH) is a new z/VM networking function announced by IBM® for z/VM Version 4 Release 4. This new feature is designed to improve the interaction between guests running under z/VM and the physical network connected to the zSeries® processor. IBM also announced IEEE 802.1Q VLAN support for z/VM Virtual Switch, z/VM QDIO Guest LAN, and z/VM HiperSockets™ Guest LAN, allowing z/VM guests to create and participate in virtual LAN configurations. This Redpaper describes how the features can be utilized by Linux guests, and how your virtual networking configurations can be greatly simplified through the use of these new functions. The following topics are covered: Introduction to VLANs z/VM Virtual Switch (VSWITCH) Configuring Linux for VLAN Comparing VSWITCH with router-based designs High availability using z/VM Virtual Switch We also describe some of our experiences with using z/VM Virtual Switch and VLANs on z/VM simulated networks.

© Copyright IBM Corp. 2003. All rights reserved.


Introduction to VLANs
Under z/VM 4.4, VSWITCH and Guest LAN include the capability to support IEEE 802.1Q Virtual LANs within the simulated network. This section introduces the concept of Virtual LANs and some of the terminology involved.

What is a Virtual LAN
A virtual LAN allows a physical network to be divided administratively into separate logical networks. In effect, these logical networks operate as if they are physically independent of each other. The concept can be difficult to describe in words, so Figure 1 shows a diagram to illustrate it.



Hub Switch 1

Switch 2

Figure 1 VLAN scenario

We make the following observations about the network in this diagram: Switch 1 is installed at one location, and Switch 2 and Hub are in another (for example, on separate floors of a building, or in separate buildings). Switch ports are represented by dark squares. The larger squares are trunk ports, the small squares are access ports.


Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.4

Definitions: A trunk port is a port that carries VLAN traffic between VLAN-aware devices such as switches. Trunk ports provide the connections that carry traffic for more than one VLAN between the switches that are used to access those VLANs. An access port is a port that is defined to a single VLAN only. Access ports provide connections for non-VLAN-aware devices, giving them access to the VLAN environment. Frames sent on access ports do not have any VLAN information attached. VLAN11 is a network that exists across both locations. This is achieved by defining trunk ports on both switches to the VLAN, and then connecting the two trunk ports. The router is a VLAN-capable device, attached to one of the trunk ports. The correct definitions in the router will provide a routing path between VLAN10 and VLAN12, and between either of these networks and the WAN (represented by the cloud). This is usually done by defining a virtual network device against the physical port on the router, linking that virtual interface to the VLAN, and enabling the interface for routing. VLAN11 has no access to any other VLAN, or the WAN. Even though VLAN11 shares access to trunk ports in both switches with VLAN12, the VLAN architecture prevents traffic from flowing between the two networks. If routing to VLAN11 is required, the Switch 1 trunk port to the router could be included into VLAN11 and a virtual interface for VLAN11 defined in the router. The only machines in the entire network that are permitted to access the server LINUXA are those in VLAN11. This is for the same reason that VLAN11 cannot access the external network: there is no routing path between VLAN11 and the other VLANs. Attention: VLANs are different from Emulated LANs (ELANs). A VLAN uses the same frame format as the underlying medium, while an ELAN often uses one network technology to carry the frames of another. An example of this is Asynchronous Transfer Mode (ATM) ELAN, which allowed Ethernet and Token Ring traffic to be carried over an ATM backbone by emulating those frame formats over ATM cells. Note that VLANs and ELANS should not be confused with the various simulated LAN technologies we use on zSeries and z/VM!

VLAN standards
Several virtual LAN mechanisms exist; most of them are proprietary and operate on a single vendor’s equipment.

Port-based VLANs
Port-based VLANs are most often proprietary solutions that function within a single vendor’s switch hardware. They provide a method of dividing a single network device (the switch) into separate broadcast domains; how the switch does this internally is platform-specific. Importantly, end stations have no way to participate in multiple VLANs because the switch isolates the devices attached to it from the VLANs in the switch.

IEEE 802.1Q VLAN tagging
IEEE 802.1Q defines a standard virtual LAN mechanism that is being implemented across equipment from many different vendors. It uses a header, called the VLAN tag, added to packets transmitted over the network. The tag contains information that allows the network to manage the separation of the different virtual LANs.
Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.4


Cisco Inter-Switch Link VLANs
Cisco Systems has a proprietary VLAN trunking mechanism called Inter-Switch Link (ISL). If your site uses Cisco networking equipment, ISL VLANs may be in use. ISL VLANs are not compatible with IEEE 802.1Q VLANs, but Cisco switches provide a function that allows mapping between ISL VLANs and 802.1Q VLANs. Important: In this document we will investigate the IEEE 802.1Q VLAN only, because this is the VLAN standard supported under Linux and z/VM. For the remainder of this paper, when we refer to VLANs, we are specifically referring to IEEE 802.1Q VLANs (unless stated otherwise).

How IEEE 802.1Q VLANs work
VLAN adds additional information to the data packet. The extra data is referred to as the VLAN tag. The tag appears immediately after the Ethernet frame header, and before the data payload of the frame. The format of the VLAN tag on Ethernet is shown in Figure 2.
(max 36)





(Ethernet encoded)


Length/Type field

(present only if required)

Octets 1













Figure 2 VLAN tag format for Ethernet

In most instances, only the Tag Protocol Identifier (TPID) and Tag Control Information (TCI) fields are used. This gives the impression that the VLAN header is only four bytes. In fact, the specification defines that additional information can be carried in the tag, including an Extended Routing Information Field (E-RIF) which would be used in source-routing environments (the Canonical Format Indicator (CFI) bit in the TCI indicates whether the E-RIF is present or not). The three-bit priority field at the start of the TCI can be used by switch equipment to prioritize frames on different VLANs. On Linux, the vconfig command has parameters that allow the priority field to be set for the VLAN being defined.


Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.4

26. Part of the function of a VLAN-capable device is to add or remove VLAN tags as required.16 Source 0. The VLAN tag is never included in a packet sent to a non-VLAN device. usually based on the learned capability of the peer device.4 5 .16 Addresses Interface queues vlan20 eth0 eth1 vlan21 vlan30 vlan40 eth2 8021q 8021q NIC driver NIC 1 NIC driver NIC 2 NIC driver NIC 3 Figure 3 Layer diagram for VLAN-capable devices Attention: This diagram is not meant to be a representation of the way that VLANs are implemented in any particular device. or it can tag the frames with a port-specific VLAN number (providing a function similar to a port-based VLAN).53.16 172. VLAN priority is used to prioritize the frames of a VLAN relative to other VLANs.26.16 172.8 172. A frame generated by a device that is not VLAN-capable is called an untagged frame when it arrives at the VLAN-capable switch.59. Still other prioritization schemes may exist. The action taken by the switch in this case can vary.15.26. IP Layer Routing 172.16 172.26. Connecting to a VLAN VLAN-capable devices attach to VLANs through the use of virtual network interfaces. like the traffic shaping facilities provided by Linux.26. The diagram in Figure 3 shows the layered relationship of the components involved in packet transmission. It is a conceptual overview only.56.Note: The VLAN priority is separate from IP priority mechanisms. Each VLAN has its own separate virtual interface. Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. Untagged frames A device does not have to support VLANs in order for it to join a network that uses them. while IP prioritization operates within the IP layers of routers.0 172. The switch can assign a default VLAN number to be assigned to any untagged frames that enter the switch.55.

0.53. on the other hand. however. as illustrated with eth1 in Figure 3 on page 5. when the packet gets to the front of the queue it is given to the NIC driver for encoding and transmission on the network with no other action. This means that the device does not generate any untagged frames when communicating with VLAN-aware devices.4. The IP address associated with eth1 is a dummy address (that is. a simulated QDIO or HiperSockets network can now provide IEEE 802. Linux. when VLAN interfaces are configured. Because eth0 is non-VLAN.4 . eth2 has an almost identical configuration to eth1. The Guest LAN simulation works in exactly the same manner as in previous z/VM releases. resulting in an untagged frame being transmitted. You can do this by using a dummy address on the base interface. Figure 3 shows eth1 configured as 0. which is commonly used for this purpose because it can never be used as a source address. As for any other interface.16 as a source address. The 802. packets may arrive at the queues for these interfaces as a result of the IP layer making a routing decision. The use of untagged frames in a VLAN environment can result in unexpected connectivity problems and potential security issues. This allows the VLAN support to send the packet to the correct interface queue.0.0. The switch that eth2 is connected to must know what to do with such frames arriving when tagged frames are expected. Note: In Linux.26. To avoid this. the kernel VLAN code processes the packet by adding the VLAN tag and passing the tagged packet to the driver for the interface that the VLAN belongs to. because the IP layer will not direct any packets to that interface for transmission (nor will the network send any packets to that address). can generate untagged frames in the way described for eth2. it must be configured to handle these frames.1Q specification refers to this kind of port as a hybrid port. the Guest LAN simulation 6 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. The IP layer. An example of what can happen with untagged frames in a VLAN environment is shown in “VLAN Isolation with untagged frames” on page 32. except that it can pass VLAN tagged frames between guests that are VLAN-aware. always ensure that network traffic specifies a VLAN ID. Here. Note: Some equipment (such as Cisco switches/routers) specify a default VLAN used for any interfaces that do not have an explicit VLAN ID coded. places the packet onto the queue for eth0. The virtual interfaces for these VLANs are vlan20 and vlan21. eth1 is an interface that supports connection to two VLANs. eth0 is a non-VLAN device. 20 and 21. a valid IP address is configured on the eth2 interface. it can act both as a trunk port to handle tagged frames.In Figure 3. see “Untagged frames” on page 5 for more on this. an address which does not match a real network configuration). When the IP layer selects 172. once a routing decision has selected eth0 as the interface for the packet to be transmitted on. If the switch that this interface connects to is VLAN-aware. the physical interface the VLAN is associated with must be provided as part of the configuration. This means that switch port does not need any configuration to support untagged frames (it can operate purely as a trunk port). except for the address associated with it. The practical effect of this is that the IP interface eth1 will not generate any traffic. For VLAN interfaces. When using VLANs over Guest LAN.1Q VLAN function to attached guests. respectively. and as an access port to handle untagged frames. VLAN support on z/VM Guest LAN With z/VM 4. the packet is not processed by VLAN.

It might seem that a VSWITCH without an OSA is just the same as a QDIO Guest LAN. External network access using VSWITCH VSWITCH provides a way to link an external network to guests under z/VM via an OSA Express. z/VM Virtual Switch (VSWITCH) z/VM Virtual Switch is an extension of the Guest LAN simulated networking function provided in earlier releases of z/VM.4. without the need to configure Proxy ARP in a z/VM TCP/IP service machine. This TCPIP service machine acts as a controller for the VSWITCH. Guests attached to the VSWITCH appear to be attached to the LAN that the OSA Express is attached to. the HiperSockets microcode on zSeries processors does not support VLAN-tagged frames.does not interfere with the VLAN tagging of network frames. Another feature added to z/VM simulated HiperSockets with z/VM 4. This means that a guest attached to a Guest LAN must be VLAN-aware in order to participate in the VLAN network. Important: At this time.3 QDIO Guest LAN. z/VM Virtual Switch can also function in a disconnected mode.4 is broadcast capability.1Q VLAN support We describe various functions provided by the new support in the following sections. the VSWITCH provides additional control over VLAN membership and handling of untagged frames (refer to “VLANs on z/VM Virtual Switch” on page 9 for more detail). TCPIP service machine “controllers” z/VM Virtual Switch uses a TCPIP service machine to act as the interface to an OSA Express network connection.4 7 . it operates almost exactly the same as a z/VM 4. This means that guests attached to a simulated HiperSockets can now use applications and protocols that use IP broadcast. We mention this mainly so that you don’t incorrectly refer to VSWITCH as a bridge when talking to the network staff. and manages the operation of the OSA Express adapter ports the VSWITCH uses. in addition to the IP multicast support added in a previous release. VSWITCH will handle only IP packets. where either an OSA port is not associated. or the associated OSA does not flow traffic to the external network. and as such does not qualify as a full Layer 2 network bridge. VSWITCH provides a function that operates much like a Layer 2 switch within z/VM. To z/VM guests. While this function looks like it bridges between a Guest LAN and the Ethernet. such as in the case of translational bridging between Token Ring and Ethernet). Because QDIO is an IP-only transport anyway. without the need for a routing function. but this is not the case. This allows you to configure your guests with IP addresses from the network that the OSA Express connects to. VLAN support applies only to simulated HiperSockets on z/VM 4. Important: Avoid using the term “bridging” to describe this function. with two important exceptions: Direct external network access via OSA Express IEEE 802. Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. bridging usually refers to the copying of an entire Layer 2 frame from one network to another (performing whatever frame translation is required on the way. this is not a limitation.

At least one controller TCPIP service machine must be eligible. otherwise. the *VSWITCH system service instructs it to activate the OSA port specified using the configuration given in the CP DEFINE VSWITCH or CP SET VSWITCH commands. The configuration allows the TCPIP stack to connect to the system service that manages VSWITCH connections. and that TCPIP service machine is eligible to act as a controller.01/0. that stack will act as the controller. Important: You must not define the OSA Express port to your controller TCPIP service machine in its PROFILE TCPIP.Important: The controller TCP/IP service machine is not involved in data transfer between the LAN and the guests on the VSWITCH. at least one TCPIP service machine must be configured to be a controller (this configuration is described in “Planning for VSWITCH” on page 10). Example 2 NETSTAT DEVLINKS output on a TCPIP VSWITCH Controller netstat devlinks VM TCP/IP Netstat Level 440 Device VSWITCHDEV Queue size: 0 CPU: 0 Link VSWITCHLINK BytesIn: 4355 Device TESTSW32320DEV Queue size: 0 CPU: 0 Type: VSWITCH-IUCV Status: Connected IUCVid: *VSWITCH Priority: B Type: IUCV Net number: 1 BytesOut: 4489 Type: VSWITCH-OSD Address: 2320 Status: Ready Port name: OSA2320 8 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. the request to add the OSA to the VSWITCH will fail. The TCPIP service machine is then able to act as a controller for a VSWITCH OSA connection. If a CONTROLLER was specified on the CP DEFINE VSWITCH or CP SET VSWITCH command. Example 2 shows the results of this command. When a TCPIP service machine is controlling an OSA Express port for a VSWITCH. or when an OSA RDEV is added for an existing VSWITCH.4 . In order for a VSWITCH to provide connectivity to a LAN. Example 1 CP QUERY CONTROLLER q controller CONTROLLER TCPCTL1 Available: YES SYSTEM TESTSW3 Controller: * CONTROLLER TCPCTL2 Available: YES Ready. the *VSWITCH system service scans its list of eligible TCPIP service machines for the one with the fewest current VSWITCH connections and selects that service machine as the controller. Using the CP QUERY CONTROLLER command. the *VSWITCH system service determines which TCPIP service machine is to be used as the controller for that OSA.01 17:12:16 VDEV Range: * VDEV Range: * When a VSWITCH with an OSA RDEV (real device) is defined. T=0. or CONTROLLER * was specified. Once the controller stack for a VSWITCH has been selected. you can see the device and link entries created for the VSWITCH in the output of the NETSTAT DEVLINKS command. The *VSWITCH system service dynamically adds the required definitions to the controller. Example 1 shows the output from the command. If no CONTROLLER parameter was given. you can see the TCPIP stacks that can serve as VSWITCH controllers.

This means that guests attached to a VSWITCH under z/VM 4.01 10:31:55 Arp Query Support: Yes Type: QDIOETHERNET Net number: 0 Yes Yes Here. When an OSA Express is attached to the VSWITCH. The link type shown here is VSWITCH-IUCV. or even with other VSWITCHes.4 9 . See “Multiple OSA Express ports” on page 21 for more information. VSWITCH can provide many options for controlling which guests receive frames on which VLANs.4 can participate in VLAN networking. VSWITCH will add the correct VLAN tag to frames as they leave the guest and remove it when the guest receives frames. The only time to set PRIROUTER for a VSWITCH is if you have a guest attached to the VSWITCH that is providing a routing function for systems attached to another network. This name is derived from the name of our VSWITCH (TESTSW3) and the virtual device number of the OSA. In the case where the guest is linked to a VLAN via VSWITCH configuration. The controller TCP/IP service machine is not involved in this data transfer. Frames are transferred directly between the OSA Express buffers and the guest on the VSWITCH. This is called VLAN filtering. There are additional considerations for OSA Express port sharing when multiple OSA Express ports are used for backup. and is described in “VSWITCH VLAN filtering”. Network data frames destined for guests attached to a VSWITCH are handled entirely by the CP VSWITCH buffer management logic. VLANs on z/VM Virtual Switch z/VM Virtual Switch supports IEEE 802.Router Type: NonRouter Link TESTSW32320LINK Broadcast Capability: Multicast Capability: Ready. You do not need to set PRIROUTER on a VSWITCH for normal operation. the frames will be sent and received untagged. This is the connection to the *VSWITCH system service. called in this case TESTSW32320DEV. VLAN operations extend between the VSWITCH and the LAN via the OSA Express. As long as you do not set PRIROUTER on your VSWITCH definition. Note: This means that the guest does not require any configuration for VLAN support. CP will act as a VLAN-aware switch and perform VLAN tag processing according to configuration. outgoing frames are transferred by the VSWITCH code from the guest’s virtual NIC to the OSA Express. you can share it with other images without restriction (even if one of the systems you are sharing with requires PRIROUTER). On a VSWITCH. T=0. Guests that do not support VLAN will send and receive frames on the VLAN that CP has been configured to connect them to. you can see two extra devices added that support the VSWITCH: An IUCV link called VSWITCHDEV. Likewise. This means that the guest need not be VLAN-aware in order to communicate to a specific VLAN.1Q VLANs. Sharing an OSA Express port with z/VM Virtual Switch The OSA Express port you use with your z/VM Virtual Switch can be shared between the VSWITCH and other guests (or LPARs). If the configuration of the VSWITCH does not specify a VLAN ID for that guest.01/0. created when the stack initializes with VSWITCH CONTROLLER ON specified in PROFILE TCPIP. Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. The OSA Express port device.

We recommend that you do not use untagged frames in your environment. untagged frames from within the VSWITCH are always passed to the guest if the destination IP address matches an address that the guest has configured. The rules that apply to packet delivery on a z/VM Virtual Switch are explained in the “Working with Virtual Networks” chapter of z/VM Virtual Machine Operation. We describe the planning and configuration steps we took to enable VLAN communication both within our z/VM system and to VLANs outside it. This is analogous to defining the VLAN membership of a trunk port on a LAN switch. if the VLAN IDs already exist in the LAN. This will extend those VLANs out of the VSWITCH into the LAN. This could result in unauthorized access to different network resources.If an OSA Express is associated with your VSWITCH. Important: The tasks described here are required only if you are using VSWITCH to connect to an external network via OSA Express. Only a virtual adapter “attached” to the correct VLAN will receive packets on that VLAN. you must perform several tasks. always configure the base interface with a dummy IP address as described in “Connecting to a VLAN” on page 5. there are no preparation tasks and you can go straight to “Configuring VSWITCH” on page 11.4 . If you are using VSWITCH only as a Guest LAN with VLAN filtering. VLAN isolation VLAN-capable devices automatically manage the separation of traffic between VLANs. 10 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. the guests on the VSWITCH will now be present on the same VLAN. VSWITCH VLAN filtering VSWITCH provides an additional feature that provides further isolation. you can control VLAN access at the VSWITCH. Only frames tagged with those VLAN IDs are passed to the guest. When you configure the port on the LAN switch the OSA Express connects to. use the CP SET VSWITCH command to grant the guest access to one VLAN ID only. Attention: There appears to be one exception to this: regardless of the VLAN IDs specified. You can nominate the VLAN IDs that a guest is allowed to “see” when you use the CP SET VSWITCH command to grant access to a VSWITCH. Using the CP SET VSWITCH command. For VLAN capable systems. you can choose to make that port belong to any of the VLAN numbers configured in the VSWITCH. Planning for VSWITCH Before you can start using a VSWITCH. For VLAN-unaware guests. Using z/VM Virtual Switch In this section we describe how we used VSWITCH on our system. SC24-5955. this OSA Express port will function like a trunk port with access to all of the VLAN IDs appearing in the VSWITCH.

01/0. all LPARs or guests must use the same name to open the OSA port. The real device address of the OSA Express adapter to be associated with this VSWITCH. The special value NONE is used to create a Virtual Switch that is not associated with an OSA Express connection. Define the VSWITCH A VSWITCH is created using the CP DEFINE VSWITCH command from a z/VM Class B userid. Connecting a guest to VSWITCH Guests connect to a VSWITCH using a simulated QDIO NIC. The key parameters (the ones you are most likely to use) of the command are as follows: switchname RDEV rdev This is the name of the VSWITCH being created. This authorizes the TCP/IP service machine to connect to the *VSWITCH system service. you can start using VSWITCH. Only the first device address from the OSA port’s address triple is entered here. you must specify more than one portname. This works the same way as for normal use of the OSA Express: when sharing the OSA. This name is used to identify this VSWITCH in later commands.01 20:13:02 HCPSWU2830I VSWITCH SYSTEM TESTSW3 status is ready. You can specify up to three addresses at this parameter. the VSWITCH CONTROLLER statement must appear in the PROFILE TCPIP for that stack.4 11 . There is an extra step involved with VSWITCH. This works exactly the same way as it does for Guest LAN. Add VSWITCH CONTROLLER to PROFILE TCPIP For a z/VM TCP/IP service machine to initialize as a candidate to control an OSA for a VSWITCH. Configuring VSWITCH Once your z/VM system is prepared. however: Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. This is used to provide backup OSA Express connections in case the first OSA fails. PORTname portname The port name to be used when opening the OSA Express adapter. If you are specifying more than one RDEV. HCPSWU2830I TCPCTL1 is VSWITCH controller. T=0. Important: PORTNAME (and its operands) must be the last parameter specified on the DEFINE VSWITCH command (this applies to the SET VSWITCH command also). with two OSA Express ports available. define vswitch testsw3 rdev 2320 2180 portname osa2320 osa2180 VSWITCH SYSTEM TESTSW3 is created Ready. The following shows the definition of a z/VM Virtual Switch called TESTSW3. The full syntax of the command is explained in “DEFINE VSWITCH” on page 39.Update the directory entry for CONTROLLER TCPIPs Any TCP/IP service machine that you want to operate as a controller for a VSWITCH must have the IUCV *VSWITCH statement added to its directory entry.

untagged frames will still be delivered to that guest. Important: When a guest’s access to the VSWITCH is restricted to a single VLAN.4 .01/0. define nic 6100 qdio NIC 6100 is created. devices 6100-6102 defined Ready.01/0. Here is an example of using the command to connect the simulated NIC at device address 6100 to a VSWITCH called TESTSW3. or allow the guest access to any VLAN (the default). T=0. T=0. SG24-6824 for more information on how to define a simulated NIC. Refer to “VSWITCH VLAN filtering” on page 10 for information on how VSWITCH controls frame flow. couple 6100 to system testsw3 Configuring Linux for the VSWITCH connection Once your userid has a simulated NIC defined and connected to the VSWITCH. Defining a simulated NIC A simulated NIC is defined on a guest using the CP DEFINE NIC command. connect your simulated NIC using the CP COUPLE command.01 14:38:10 The command syntax of the CP SET VSWITCH command is shown in “SET VSWITCH” on page 39. SC24-6036. Note: A simulated NIC of type QDIO can be connected to either a QDIO Guest LAN or a VSWITCH. The GRANT option allows you to restrict the guest to particular VLANs.4 Virtual Machine Operation.guests attaching to a VSWITCH must be granted access to the VSWITCH using the SET VSWITCH command. Refer to the section on creating a simulated NIC in Linux on IBM ^ zSeries and S/390: Large Scale Linux Deployment.01 14:38:01 set vswitch testsw3 grant lnxrh2 vlan 201 Command complete Ready. an excellent new chapter called “Working with Virtual Networks” is available in z/VM Version 4. Also. T=0.01 13:46:11 Attaching the simulated NIC to the VSWITCH Once your simulated QDIO NIC is created. 12 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.01/0. Granting access to a VSWITCH using SET VSWITCH Access to the VSWITCH is controlled by the GRANT and REVOKE options of the CP SET VSWITCH command. Configuring your Linux system for attachment to a VSWITCH is done in the same way as for a QDIO Guest LAN connection. set vswitch testsw3 grant lnxsu6 vlan any Command complete Ready. This example shows the use of the CP DEFINE NIC command to define a simulated QDIO NIC at device address 6100. you can IPL your Linux installation.

Note: The IBM Redbook Linux on IBM ^ zSeries and S/390: Large Scale Linux Deployment. Configure a VLAN interface. Load the IEEE 802. you need the following: Linux kernel 2. but the vconfig program was available in a prepared RPM package that came with the distribution (the vlan package). The steps involved in starting a connection to a VLAN from a Linux guest are: Configure the VLAN. On a VSWITCH. It provides a new way to define simulated NICs as part of a guest’s directory definition. Note: On our SuSE systems. rather than dynamically defined by CP when the NIC is created. or OSA Express adapters that support VLAN.1Q VLANs (the module 8021q. NICDEF provides some additional options not available when you use the SPECIAL directory control statement. Activate the physical interface.14 or higher. you do need to ensure that your guest has authority to use the VLAN you intend to use.1Q VLAN module. or an earlier kernel with the IEEE 802.o was present). Configure the VLAN If your VLAN configuration exists entirely within a VSWITCH or a z/VM simulated HiperSockets. This would be useful if you are using a protocol that relies on the MAC address of a host. NICDEF directory control statement NICDEF is a new directory control statement added to z/VM Directory Maintenance (DirMaint) with z/VM 4. and the CP SET VSWITCH command to change the guest authorization. Bring up the VLAN interface. there is no VLAN configuration required. such as DHCP. SG24-6824 gives a description of how to set up Linux for Guest LAN attachment (in the section “Configuring a VM Guest LAN in a Linux guest”). HiperSockets Guest LANs.1Q VLAN patches. Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. The options available on the DIRMAINT NICDEF command are shown in “DIRMAINT NICDEF” on page 40. the first six are set using the new MACPREFIX option on the VMLAN statement in SYSTEM CONFIG. however. MACID specifies the last six hexadecimal digits of the MAC address. This allows the MAC address allocated to a simulated NIC to be defined statically. Configuring Linux for VLAN This section describes the process to start using VLANs in your Linux systems. VLAN configuration process on Linux To connect to VLANs from a Linux system. whether on VSWITCH. Most relevant to Linux guests is the MACID parameter. to configure the VLAN virtual interfaces.4 13 . then we show you how you can do the steps automatically using the VLAN support included in SuSE SLES8. First we describe the steps that take place to start a VLAN interface.4. Use the CP QUERY VSWITCH command to check VLAN authority.4. not only was the kernel already prepared to support 802. A user-space utility called vconfig.

Apart from creating and deleting virtual interfaces. If you are not going to be using the physical interface to attach to an IP network (that is. however.) character in the interface name. Miller <davem@redhat.0. On SuSE SLES 8. You can use the address 0. Configure a VLAN interface You create your VLAN virtual interface using the vconfig command. changing the relative priority of packets for different VLANs.1Q VLAN Support v1. and our configuration was very easy).0. you can configure the interface in the same way as usual.1Q VLAN module VLAN support is usually built as a module called 8021q. you will see the following messages in your system log or dmesg output: 802. have a problem with the period (. the network configuration scripts worked correctly. You can specify the priority of packets on your VLANs. either directly or through a VSWITCH. or you need to set or remove VLAN ANY authority. You can also change the default way in which the virtual devices are named. the 8021q. and al your IP addresses and network traffic will be associated with VLANs) you configure a dummy’ IP address on the physical interface. the vconfig command allows you to control the operation of the VLAN interfaces.0 as a valid dummy address (in fact. If you are going to use a default VLAN or generate untagged frames. If you will be connecting to a VLAN in your switch hardware.0.0. Load the IEEE 802. Use the insmod or modprobe command to load the module before attempting any operations to configure or use VLANs. we used the ifcfg-ethX file in /etc/sysconfig/network to define the address for our base interface.o module uses the DEV_PLUS_VID_NO_PAD format.7 Ben Greear <greearb@candelatech. Refer to “Switch configuration” on page 24 for aspects of switch configuration that you may need to investigate.0. the interface simply gives you a connection to the network. you can only change this by first revoking the current authority and granting the new authority.o. so either the VLAN_PLUS_VID or VLAN_PLUS_VID_NO_PAD formats must be used. 14 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.4 .Important: If the guest is already authorized to a VLAN and you need to change to a different> Activate the physical interface The physical interface used to connect to a VLAN must be activated prior to configuring any VLAN interfaces. The following shows how to load the module using the modprobe command: # modprobe 8021q You do not need to load a module if your kernel includes the VLAN support in the kernel (rather than in a module). Even when we used a dummy address. ensure that the switch configuration supports the VLAN you intend to connect> All bugs added by David S. When the module is loaded. By default. the IP configuration scripts seemed to recognize what we were doing when we used 0. Some programs.

you can use the ip command to configure the interface instead.10.255. Refer to “vconfig” on page 40 for an explanation of the vconfig command syntax.29 netmask 255.4 15 . The following command will configure the VLAN 10 virtual interface on eth0 with the IP address 192. Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. and have rewritten their network configuration scripts using iproute2 instead of the traditional IP configuration commands such as route and ifconfig. the system will set up your VLAN interfaces for you. configuring VLANs has been a manual process because the network configuration scripts provided with most distributions were not “VLAN-aware”.Important: One program that has a problem with the dot in the interface name is iptables.29: # ifconfig eth0.168. traffic shaping. Configuring VLANs on SuSE SLES8 Since VLAN support is a fairly recent addition to Linux. however. Bring up the VLAN interface The VLAN interface added using vconfig supports the usual interface configuration commands such as ip and ifconfig. Tip: The vconfig usage text (shown in “vconfig” on page 40) shows you what your VLAN interface names will look like for different settings of the VLAN name format. The following command will create a virtual interface for VLAN 10 against the eth0 interface: # vconfig add eth0 10 If the default VLAN name format of DEV_PLUS_VID_NO_PAD is set. so as long as you can use the right names for your VLAN interfaces.10 # ip link set eth0. which provides the ip command. you will not be able to use the default name format.29/24 dev eth0. If you plan to use any iptables commands against your VLAN interfaces.168.10 192. In addition.168. iproute2 provides the facilities to control Linux advanced routing features like multiple route tables.10.10. Many distributors now include it by default. You configure the virtual interface in the same way as any other network interface on your system. You would use these ip commands to get the same result as the ifconfig command above: # ip addr add 192. allows a great degree of flexibility in configuring the Linux IP stack. SuSE has set up their network scripts.10. the interface to the netfilter code in the Linux kernel.10 up Tip: The iproute2 utility by Alexey Kuznetsov.255. the command would create a VLAN virtual interface called eth0. and policy routing.0 up If the iproute2 utility is installed on your system. The iptables program is used to set up firewall and network address functions in your Linux system.

0' NETWORK='192. The contents of this file should be similar to Example 3. so it is unlikely that this restriction will cause a problem for you.201. If you need the ability to have the same VLAN numbers on different real interfaces representing different VLANs. the path to the vconfig executable may be incorrectly specified as /sbin/vconfig instead of /usr/sbin/vconfig. The STARTMODE field works the same as for physical interfaces. this allows you to add your VLAN interfaces to your network configuration and have them activated automatically at boot-time. If the file is named incorrectly (or more accurately: if the name you pass to the ifup script.168. Example 3 ifcfg-vlan example ETHERDEVICE='eth4' BOOTPROTO='static' STARTMODE='onboot' IPADDR='192. This is used to instruct vconfig which physical interface to configure the VLAN on.201. Fixing the VLAN script Unfortunately you may find a bug in the script used to configure the VLAN interface.255. the details shown in Example 3 would be contained in a file called /etc/sysconfig/network/ifcfg-vlan10. the SuSE network configuration processing will not call the correct script to configure the interface. you will have to update the script to show the correct path to the vconfig program. In the /etc/sysconfig/network/scripts/ifup-802. Once you have prepared your ifcfg file. is not correct).255' The ETHERDEVICE field is additional for VLAN configurations.109' NETMASK='255.1q script. The following command will start the vlan10 interface: # ifup vlan10 16 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. This means that VLAN numbers have scope across the kernel.4 .Restriction: In addition to the naming issue.0' BROADCAST='192. As an example. VLAN numbers should be allocated universally.255. Configuring your VLANs To configure a VLAN on SLES8. rather than just per interface. Important: The name of the ifcfg file you create is critical. This will cause the script to fail.168. if you are configuring VLAN ID 10. create a file in the /etc/sysconfig/network directory called ifcfg-vlanX. you start your VLAN interface by issuing the ifup command with the VLAN interface name as the parameter. Until SuSE has a fix for this.168. which has to match the last part of the ifcfg file name. the configuration requires that the vconfig per_kernel option be set to on. you will not be able to use SuSE’s configuration method and will have to create your own.201. replacing X with the VLAN number you are configuring.

See “Activate the physical interface” on page 14 for information on this. you can use it to communicate with other guests on the same VLAN.240 ms Note: In “z/VM Virtual Switch failover” on page 28.4 17 .109).168. If you included “STARTMODE=onboot” in the ifcfg-vlanX file for your VLAN interface. if a guest authorized to connect to only one VLAN generates untagged frames.762 ms 2 192. Using traceroute. two guests on the same VSWITCH but configured on separate VLANs can only reach each other via a router. This can provide a little more information than traceroute. a utility that will show you the path to other devices in the network. Making your VLAN configuration persistent If you use the supplied configuration processes on SuSE SLES8 (discussed in “Configuring VLANs on SuSE SLES8” on page 15). An example traceroute command execution is shown in Example 4.677 ms 0.098 ms 1. you do not need to do anything else to have your VLAN connections start at reboot time.109 1. The difference between Guest LAN and VSWITCH is in the handling of untagged frames. You can also use traceroute. you can verify that your VLAN configuration is working as you expect.026 ms 1.168. If your VSWITCH is linked through an OSA to stations on the LAN.168. In fact. and your LAN is configured with the same VLAN.109 traceroute to 192. Refer to the discussion there for more details. 40 byte packets 1 192. and using traceroute will help you verify this. the VSWITCH tags them with the VLAN ID the guest is authorized to. we use the -R switch on the ping command to display the route taken both to and from the destination host. 30 hops max.805 ms 0.201. If you are not using SuSE SLES8.200. you will have to write a script that implements the steps outlined in “VLAN configuration process on Linux” on page 13. the VSWITCH applies this VLAN tag to all the frames the guest generates.200.200. Example 4 traceroute command example # traceroute 192.168.109 (192. Comparing VLANs on VSWITCH and Guest LAN Configuring Linux guest connections to VLANs on a z/VM Guest LAN is identical to the process used on VSWITCH.168. or you cannot use SuSE’s process because of your configuration requirements. Testing the configuration Once your virtual interface is up. the system network initialization scripts will do all the VLAN configuration for you at system startup. you can now reach LAN devices on that VLAN as well. Utilities The ping command is a simple command you can use to verify connectivity.Important: The physical interface (the interface specified in the ETHERDEVICE parameter in the interface configuration file) must be activated in order for this to function. In VSWITCH. For example.1 0.200. Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. restricting that guest to generating frames for its authorized VLAN only1.

18 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. Guests that are VLAN-aware must be configured for matching VLAN IDs in order to communicate on their VLAN.1 Guest LAN 192. Guests with no VLAN support generating untagged frames will not have their frames modified either. multiple OSA ports.13 sD Master VRRP Virtual IP: 192.12 sC .42. the Guest LAN will carry those frames unaltered and other guests that are VLAN-aware will be able to process them. No VLAN filtering takes place.0/24 Backup . Frame processing for VLAN-aware and non-VLAN-aware guests occurs as follows: When guests that are VLAN-aware put tagged frames onto the Guest LAN.8 . This configuration provides a high level of availability through its use of multiple router guests. sA . and dynamic routing facilities. Non-VLAN-aware guests will not be able to process these frames. will give you more information about this configuration.42.9 rA z/VM OSA OSPF rB OSA Switch Switch Router Router Figure 4 Connectivity based on Linux routers 1 This is similar to the configuration of a “default VLAN” on some LAN switches.4 .168. Comparing VSWITCH with router-based designs Figure 4 shows a virtual network configuration based on Linux virtual routers.168. REDP3657. Note: The Redpaper Linux on IBM ^ zSeries and S/390: Virtual Router Redundancy Protocol on VM Guest LANs.The z/VM Guest LAN does not process or modify VLAN tags in any way.11 sB . and both VLAN-aware and non-VLAN-aware guests will be able to process the packets (as long as the VLAN-aware guests have a network definition that supports untagged frames).10 .

4 19 . not shown here is that the routers will be participating in the dynamic routing environment used for the rest of the LAN. The Linux guests consume memory.168. OSPF) generates network traffic and consumes processor resource. you remove the overhead of your Linux router guests. we show the two network routers participating in VRRP to provide a redundant gateway service for the Linux guests. Importantly. Also. Note: In Figure 5.0/24 z/VM TCPIP1 OSA TCPIP2 OSA Switch Switch Master VRRP Virtual IP: 192. Figure 5 shows an example configuration based on VSWITCH that provides a high degree of availability. With VSWITCH. Running the networking protocols that provide the routing failover (VRRP.42. it has a few drawbacks in a z/VM environment. as well as the network traffic generated by the protocols used to provide high availability in the network.While this configuration provides excellent availability for the Linux guests. sA . it does not have to be VRRP. disk and CPU resource just by being active. OSA ports must be activated with the PRIROUTER setting. a highly available network configuration can be made without the resource overhead of virtual routers. the function of providing network availability is moved into network components. like Cisco Hot Standby Router Protocol (HSRP).10 .168.12 sC . may be just as effective in your environment.42.11 sB . While we strongly recommend that you use some kind of redundant gateway facility. where it can be more consistently managed with the rest of the enterprise network function. Other such protocols. Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.1 Backup Router Router Figure 5 Connectivity based on VSWITCH By using VSWITCH in this configuration. This stops you from sharing an OSA ports between multiple routers.13 sD VSWITCH 192.

For example. Define the new simulated NIC to the guest. Important: For all migrations to z/VM Virtual Switch. and copy the new configuration files into place from the console. switch the guest’s NIC definition to QDIO. Alternatively (if you’re comfortable with the Linux console). and the CP commands that define and modify them can be run by Class B users connections to a single shared-access transport facility. we recommend setting up the new VSWITCH in parallel with the existing HiperSockets Guest LAN. For this migration. you simply change your existing CP SET LAN commands into CP SET VSWITCH commands. the default route for the guests should specify a LAN router rather than a z/VM TCP/IP service machine or Linux router guest. If your current Guest LAN is defined as UNRESTRICTED. remember to verify that the guest’s routing definitions are correct after the change is made. because you cannot use the same simulated NICs for HiperSockets Guest LAN and VSWITCH. they are all owned by SYSTEM. You cannot define a transient z/VM Virtual Switch. you will need to consider changing to persistent LANs. Restriction: If you use transient Guest LANs in your current configuration (LANs defined and owned by z/VM general users instead of SYSTEM). HiperSockets Guest LAN to z/VM Virtual Switch This migration is a little more complex. because the topology of the guest connectivity changes from a set of point. you can prepare new configuration files in advance. This change will require a corresponding change in the guest’s TCP/IP configuration.Migrating to z/VM Virtual Switch Changing your current guest networking configuration over to z/VM Virtual Switch can be achieved with a minimum of reconfiguration effort. vCTC or IUCV to z/VM Virtual Switch Migrating either vCTC or IUCV topologies to z/VM Virtual Switch is a complicated process. you can simply replace them with a z/VM Virtual Switch. If your current Guest LAN is RESTRICTED. and complete the configuration of the new VSWITCH interface while the HiperSockets Guest LAN is still available. QDIO Guest LAN to VSWITCH If you are currently using persistent QDIO Guest LANs in your z/VM environment. You will need to change the NIC definition for the guest from TYPE HIPER to TYPE QDIO. if your VSWITCH connects to an OSA Express. you will need to add appropriate CP SET VSWITCH commands to give your guests authority to connect to the VSWITCH. 20 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.4 . All your guests will connect to the VSWITCH in the same way as the Guest LAN they connect to now. Setting up the VSWITCH and simulated NICs in advance is definitely recommended for this migration.

This means that you can set up additional TCPIP service machines with no specific IP configuration to serve as backup controllers for your VSWITCH configuration. For this reason. Example 6 on page 24 shows the output from the command. You can specify multiple OSA Express ports using extra values on the RDEV parameter to the CP DEFINE VSWITCH and CP SET VSWITCH commands. Multiple CONTROLLER TCP/IP service machines If you have only one TCPIP service machine and it is logged off or fails. Multiple OSA Express ports When you specify more than one RDEV on the CP DEFINE VSWITCH command. You can make access to your z/VM guests highly reliable by configuring the redundancy features of VSWITCH combined with LAN-based high availability. Using the CP QUERY CONTROLLER command. or HOME addresses. The syntax of the DIRMAINT NICDEF command is shown in “DIRMAINT NICDEF” on page 40. or the like. with little configuration or management overhead. High availability using z/VM Virtual Switch z/VM Virtual Switch can be used to create highly-available connectivity for your Linux guests under z/VM. We describe a way of doing this in “Configuring controller TCP/IP service machines” on page 22. redundant OSA ports are made available to your VSWITCH in case network access through the first port is impaired. and multiple TCPIP controller service machines for some software redundancy. When you have more than one TCP/IP service machine running in your z/VM system. you can ensure that your z/VM guests stay linked to the external network when failures occur. these stacks can all be set up to become a VSWITCH controller.4 21 . NICDEF provides additional parameters to define simulated NICs. consider using the new NICDEF statement instead. and shows that our system currently has three TCPIP service machines ready to operate as controllers. You can define multiple OSA Express adapters for hardware redundancy. Planning redundant configuration for VSWITCH Some planning is required to ensure that your VSWITCH connectivity will survive outages. They do not need DEVICE and LINK statements. Up to three RDEV values can be Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. all your external network connectivity via a VSWITCH fails also. As long as your LAN switch is configured appropriately. Tip: z/VM TCP/IP stacks used as controllers do not need to have any connectivity defined of their own. including: Specifying the virtual channel path identifier (CHPID) the simulated NIC will use (most useful for simulated HiperSockets NICs for use on z/OS guests) Specifying the MAC address for the simulated NIC NICDEF is discussed in “NICDEF directory control statement” on page 13.Virtual NIC definition If you currently define your guests’ Guest LAN connections using the SPECIAL directory control statement. you can see the TCPIP stacks that can serve as VSWITCH controllers. it is highly recommended to have at least one additional TCPIP service machine configured as a controller to provide continued connectivity.

The configuration we tried was a success. if the PRIROUTER setting appears in the port configuration for both your VSWITCH and that of the guest or VSWITCH using the port. and they must all be configured to trunk the correct VLANs to the VSWITCH. This means that your backup ports can be in use on other guests. to act as controllers. and this conflicts with or does not match the configuration you need for the VSWITCH connection. This configuration would allow us to easily and quickly start additional controller machines from the same configuration. Demonstration: “z/VM Virtual Switch failover” on page 28 shows the results of the testing we performed with these availability configurations. we had two OSA Express ports at devices 2320-2322 and 2180-2182. the switch ports you connect your OSA Express adapters to must all be defined as trunk ports.provided. some or all of your guests may fail to connect properly if your main OSA Express connection fails. and the detail of how it was set up appears in the following paragraphs. the port cannot be used for VSWITCH backup. They each had their own 191 disk. The second OSA was the one used for direct access by other Linux guests to the LAN switch. which they link to as their 198 configuration disk. Importantly. We also defined two extra TCPIP service machines. In the following paragraphs we will describe how to build it. A very important configuration requirement when defining multiple OSA Express ports for backup is that the switch configuration of the two ports must match. Configuring high availability with VSWITCH Figure 5 on page 19 shows an example of a high availability configuration. Configuring controller TCP/IP service machines You can use your main TCPIP service machine as a VSWITCH controller. meaning up to three OSA ports can be provided to connect the VSWITCH to the LAN. If the primary system using the OSA Express port requires a certain configuration in the switch. allocated as TCPMAINT 298 (the same size as TCPMAINT 198). In our lab configuration.4 . 22 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. Minidisk configuration The controller service machines were given a separate configuration disk. Important: Only one port is used for communication between the VSWITCH and the LAN at any one time. If you don’t do this. but we decided to try setting up a separate configuration for a dedicated controller service machine. named TCPCTL1 and TCPCTL2. However. or on other VSWITCHes. you will not be able to use that port as a backup (your VSWITCH will not be able to set PRIROUTER with the other connection active on the port). As discussed in “Planning redundant configuration for VSWITCH” on page 21. to make your VSWITCH connectivity redundant. if you are extending VLANs between your VSWITCH and the LAN. Additional ports defined for the VSWITCH are not connected until required. you define multiple controller TCPIP service machines and provide multiple OSA Express ports to your VSWITCH. Important: This may also prevent you from sharing an OSA Express port for backup purposes.

We chose the latter method. apart from the VSWITCH CONTROLLER ON statement required to allow the TCP/IP stack to initialize as a VSWITCH controller. as shown in Example 5.TCPCTL1 :TYPE. and it needs to be different on each of your controller service machines.SERVER :DISKWARN.STACK Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. Attention: While we did not need to specify the LDEV range in the VSWITCH CONTROLLER parameter in our configuration. Important: If you specify the LDEV parameter. Example 5 TCPCTL1 DTCPARMS file :NICK. you will need to have a separate 198 disk for each one. We used the following commands: FORMAT 298 K COPYFILE * * D = = K (OLDDATE # answer the prompts for volume label and confirm Directory entries We added the controller service machines to the z/VM directory as copies of the main TCPIP userid. we used the default operation. SYSTEM DTCPARMS Every TCP/IP service machine in your z/VM system needs to have DTCPARMS configured.Setting up the configuration disk for the new service machines involved formatting the new minidisk and copying all the files from the configuration disk for the main TCP/IP service machine. allowing the OSA Express to be defined to the TCP/IP machine with the same logical device numbers as the real OSA device addresses (RDEVs).NO :CLASS. Our DTCPARMS file was very simple. This configuration allowed us to have an identical PROFILE TCPIP on all the controller service machines and share the configuration disk between all the controller service machines we set up. Allocate additional TCPMAINT minidisks as copies of the TCPMAINT 198 disk as described in “Minidisk configuration” on page 22 (using different device numbers for each. of course) and update each guest’s directory entry to link to the correct minidisk. the link command for the machines’ 198 disk must specify TCPMAINT 298 instead of TCPMAINT 198. or use a machine-local DTCPARMS file containing the definitions on a local minidisk to the guest. The IUCV *VSWITCH directory entry is required to allow them to connect to the system service that manages VSWITCH connectivity to OSA Express adapters. Also. as follows: LINK TCPMAINT 0298 0198 RR PROFILE TCPIP The PROFILE TCPIP for our controller service machines requires no special customization.4 23 . You can add the definitions to the SYSTEM DTCPARMS file. creating TCPCTL1 DTCPARMS and TCPCTL2 DTCPARMS files on the controller service machines’ 198-disk. We did not see a need in our configuration to restrict the machines to a range of LDEV (logical device) values. you may need to do so to prevent the *VSWITCH system service from using device numbers that are already in use on your controller service machines. This may be particularly relevant if you are using your production TCP/IP service machine as a VSWITCH controller.

we specified multiple values on the RDEV and PORTNAME parameters to provide access to our two OSA ports. you will need to ensure that your switch configuration reflects the connectivity you want.4 .01/0.01 20:17:43 Active: 3 MFS: 8192 5 VDEV: 2320 MAXCONN: INFINITE ACCOUNTING: OFF QUEUESTORAGE: 8 Switch configuration To make sure you get the connectivity you expect in your environment. Example 6 CP QUERY CONTROLLER q controller CONTROLLER TCPCTL1 Available: YES SYSTEM TESTSW3 Controller: * CONTROLLER TCPCTL2 Available: YES Ready. Example 8 shows this output. Example 6 shows the results we obtained after our controller service machines initialized. Remember: PORTNAME must be specified last.01/0. 24 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. last on DEFINE or SET VSWITCH commands. Example 7 DEFINE VSWITCH results define vswitch testsw3 rdev 2320 2180 portname osa2320 osa2180 VSWITCH SYSTEM TESTSW3 is created Ready. Issue the CP QUERY CONTROLLER command to see that your service machines have initialized and that they have registered as controllers.Start the controller service machines Once configured. To get information about the z/VM Virtual Switch we just created. Example 8 QUERY VSWITCH results query vswitch testsw3 VSWITCH SYSTEM TESTSW3 Type: VSWITCH PERSISTENT RESTRICTED NONROUTER State: Ready CONTROLLER: TCPCTL1 IPTIMEOUT: PORTNAME: OSA2320 RDEV: 2320 PORTNAME: OSA2180 RDEV: 2180 Ready.01 17:12:16 VDEV Range: * VDEV Range: * Configuring multiple OSA Express ports Remember. T=0. Note: Our testing environment used a Cisco Catalyst 6509 switch. We will illustrate the settings you need to investigate with configuration statements we used in our switch.01 20:13:02 HCPSWU2830I VSWITCH SYSTEM TESTSW3 status is ready. When we defined our z/VM Virtual Switch using the CP DEFINE VSWITCH command. Example 7 shows the results of our DEFINE VSWITCH command.01/0. we can issue the CP QUERY VSWITCH command. the new controller service machines can be logged on using the XAUTOLOG command. Important: Remember to add your new service machines to your automation procedures to ensure that they start automatically after an IPL of z/VM. T=0. T=0. Some of the things you need to look at or check are described here. HCPSWU2830I TCPCTL1 is VSWITCH controller.

itso6509 (enable) set vlan 202 type ethernet mtu 1492 Note: Depending on your logging level.-----.---------. using IEEE 802.------.4 25 . The following command defined VLAN 202 on our switch.1Q tagging.1025-4094 This command sets port 7 in module 2 to be a trunk port.-------- Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. You can use the show vlan command to display attributes of your VLAN configuration.2/11 15/1 VLAN Type SAID MTU Parent RingNo BrdgNo Stp BrdgMode Trans1 Trans2 ---. VLAN numbers 1 through 1005 and 1025 through 4094 will be trunked on this port (this is the Cisco default). some messages may appear when you enter a configuration command.---. Important: When using VLANs in z/VM Guest LANs.-------------------------------. VLANs are defined using the set vlan command. Example 9 gives an example of this.--------.Setting up a trunk port If you are using VLANs within your VSWITCH. Vlans ---. On our Cisco switch.-----. Example 9 Displaying VLAN status on a Cisco Catalyst switch itso6509 (enable) show vlan 202 VLAN Name Status IfIndex Mod/Ports. we defined a trunk port with this command: set trunk 2/7 on dot1q 1-1005.---------. however. you will need to configure the switch port your OSA Express attaches to as a trunk port.2/7.----. Defining a VLAN Your LAN switch must include a definition for the VLAN you want to create. you do not need to pre-define the VLANs in your z/VM configuration. This would be useful to minimize unnecessary packet flow into the VSWITCH if you have many VLANs defined in the switch network but only some of them are represented in the VSWITCH.-----. This ensures two things: Frames sent to the OSA Express are VLAN tagged Frames for your desired VLANs are sent to the OSA Express You can configure the trunk port to carry a specific range of VLANs.---------. In Cisco CatOS. and you want to link the VLANs inside z/VM with VLANs in the LAN. We specified an MTU of 1492 bytes to correspond with the MTU used on the OSA Express and VSWITCH adapters.-----. you must grant access to VLAN IDs as required.-----202 enet 100202 1492 0 0 VLAN MISTP-Inst DynCreated RSPAN ---. If you are using VLAN filtering.----.-------.-----------------------202 active 114 2/3.

Adding ports to a VLAN One of the first things you might want to do is configure certain ports on your switch to access certain VLANs. which provides a routing component.-------------------. Example 10 Displaying port information on a Cisco Catalyst switch itso6509 (enable) show port 2/3 * = Configured MAC Address Port Name Status Vlan Duplex Speed Type ----. Example 10 gives a sample of this display.> Port Last-Time-Cleared ----. 06:56:44 Idle Detection --------------itso6509 (enable) We have truncated some of the output of this command.. or it may be a software or hardware component of the switch.-----.---------. The Cisco Catalyst switch in our lab was equipped with the Multilayer Switch Feature Card (MSFC).-----------2/3 connected 202 a-full a-100 10/100BaseTX <. itso6509 (enable) set vlan 202 2/3 The show port commands will display various information about your port status.--------.-------.-------..------------.-------.4 .-------. The following command will attach port 3 on module 2 to VLAN 202. We used the CatOS set vlan command to assign switch ports to VLANs.---------.202 - static disabled itso6509 (enable) There are many other configuration settings for VLANs available. Refer to your Cisco documentation for more detail. The router may be a physical device with attachments to ports on the switch...----. This is shown by the <.-------------------------2/3 Sun Jul 6 2003.> symbols in the output.> Port Security Violation Shutdown-Time Age-Time Max-Addr Trap IfIndex ----. Routing to and from your VLAN Stations attached to different VLANs cannot communicate to each other without a router.------2/3 disabled shutdown 0 0 1 disabled 12 <... We used this routing function to provide access between our VLANs. 26 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.

one per line.202.168. 0 underruns 0 output errors.85f3. You can display the status of your VLAN virtual interface using the show interface command.7302 (bia 0007. The diagram in Figure 6 on page 28 shows our network testing configuration. output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/0 (size/max/drops/flushes). 28982480 bytes. Refer to your Cisco documentation for more information.1/24 MTU 1500 bytes. address is 0007. Total output drops: 0 Queueing strategy: fifo Output queue :0/40 (size/max) 5 minute input rate 0 bits/sec. line protocol is up Hardware is Cat6k RP Virtual Ethernet. 0 CRC.Configuring a connection to a VLAN for routing requires adding a virtual interface to the MSFC representing a connection to the VLAN.4 27 . The interface vlan command is used to create a VLAN virtual interface. Example 12 shows a sample of this. txload 1/255. output never. reliability 255/255. 0 packets/sec 5 minute output rate 0 bits/sec. Experiences This section describes some aspects of VSWITCH and HiperSockets VLAN support that we encountered while writing this paper.255.1 255. Example 12 Interface status for a MSFC VLAN virtual interface itsores1>show interface vlan202 Vlan202 is up. 0 ignored 465 packets output.7302) Internet address is 192.0 itsores1(config-if)#^Z itsores1# Additional commands are required to enable routing in the MSFC. 0 output buffers swapped out itsores1> There are many other commands you can use to get information about your routing configuration. 0 overrun. Refer to Cisco documentation if you need assistance with these. 38350 bytes.255. and the ip address command is used to assign an IP address to the virtual interface. DLY 10 usec.168.202. and to set up a dynamic routing protocol if you use one. 0 no buffer Received 556777 broadcasts. loopback not set ARP type: ARPA. BW 1000000 Kbit. 0 runts. Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. itsores1(config)#interface vlan 202 itsores1(config-if)#ip address 192. rxload 1/255 Encapsulation ARPA. Example 11 shows the commands we used to add a VLAN interface to the configuration of the routing function in our switch. End with CNTL/Z. Example 11 Configuring MSFC for VLAN routing itsores1#configure terminal Enter configuration commands. 2 interface resets 0 output buffer failures. 0 frame. 0 packets/sec 556993 packets input. ARP Timeout 04:00:00 Last input 00:00:01.85f3. 0 throttles 0 input errors. 0 giants.

35 LNXSU3 LNXSU2 HiperSockets Guest LAN ITSO — 1 2 201 202 LNXSU1 TCPIP VSWITCH TESTSW3 Linux Guests 192. we represented the VLANs in a switch (including z/VM Virtual Switch) as labelled rectangles within the representation of the switch. on the other hand.100. that there was no alternative path for the connection to travel over). A particular station or guest’s membership of a VLAN is indicated by a solid circle in the rectangle for the VLAN intersecting the connection from the station. we tried various ways to disrupt the communication through the OSA Express port to the LAN.168. We used the ping command from one of the Linux guest machines on our VSWITCH to a PC attached to a port on the LAN switch.0/24 Switch/Router Routing Cisco Catalyst 6509 Windows Test Station 192.36 .46 . For example.168.202. was granted access to VLAN 201 only. 28 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. LNXRH2. We made sure that the LAN PC was only reachable via the VLAN link via the VSWITCH (that is. the guest LNXSU7 belongs to VLAN . as well as the relationship between the VSWITCH OSA Express port and the controller TCPIP service machine.45 LNXSU7 .0/24 VMLINUX HiperSockets 0 2211 .168.37 LNXSU4 LNXRH2 . The diagram also shows a HiperSockets Guest LAN with a single VLAN.200. z/VM Virtual Switch failover With the configuration as described in “Configuring high availability with VSWITCH” on page 22. In the VSWITCH. the ability to generate and receive untagged frames is shown by membership to the ‘—’ VLAN.38 .12.44 LNXSU6 . and verified this by using the -R switch on the ping command. but because we granted it access to VLAN ANY it can also generate and receive untagged frames.0/24 Figure 6 Test network configuration In this diagram.0/24 OSA VLANs OSA 1 2 201 202 Linux Test Servers 192.

109 64 64 64 64 64 64 64 bytes bytes bytes bytes bytes bytes bytes from from from from from from from 192.109 : 56(124) bytes of data.01/ ms icmp_seq=12 ttl=64 time=0.100) from 192.100 ( ms icmp_seq=3 ttl=64 time=0.100: 192.200.100 ping statistics --13 packets transmitted.200.168. so that when the reply is received by ping. if your round trip exceeds nine hops.168.168. you will only see the first nine.200. HCPSWU2830I TCPCTL1 is VSWITCH controller.100: 192.200. 38% loss.592 ms icmp_seq=4 ttl=64 time=0.100: 192.168. but the response is much quicker and you obtain round-trip data (traceroute will only tell you the path from source to destination. only nine hops can be counted using ping -R.100 PING 192.200.disable here \ unsolicited / messages / Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.168. The effect is similar to traceroute. We used the set port disable command in the Cisco Catalyst switch to shut down the switch port.168.100: 192.200.597 ms icmp_seq=11 ttl=64 time=0. Example 14 z/VM commands and messages around switch port deactivation q vswitch testsw3 VSWITCH SYSTEM TESTSW3 Type: VSWITCH Active: 1 PERSISTENT RESTRICTED NONROUTER MFS: 8192 State: Ready CONTROLLER: TCPCTL1 IPTIMEOUT: 5 PORTNAME: OSA2180 RDEV: 2180 PORTNAME: OSA2320 RDEV: 2320 VDEV: 2320 Ready.623 ms (same (same (same (same (same (same (same route) route) route) route) < disable here route) route) route) --.168.168. T=0.01 15:45:31 HCPSWU2830I VSWITCH SYSTEM TESTSW3 status is devices HCPSWU2830I TCPCTL1 is VSWITCH controller.100: icmp_seq=1 ttl=64 time=1. it can display the route taken by the packet.100: icmp_seq=2 ttl=64 time=0.4 29 .200.14 ms RR: 192.100: 192. HCPSWU2830I VSWITCH SYSTEM TESTSW3 status is ready.200.168.200.before disable MAXCONN: INFINITE ACCOUNTING: OFF QUEUESTORAGE: 8 attached. time 12006ms rtt min/avg/max/mdev = and sometimes the return path is different). Deactivate the switch port in the LAN switch This failure is detected as a loss of signal or cable pull on the OSA Express.679/1.613 ms icmp_seq=5 ttl=64 time=0. Example 13 shows the results or our ping test.178 ms lnxsu6:~ # You can see that after a short period of time.100 192.100 192. the connection was transferred to the other OSA.100: 192.168. we issued the commands and received messages as shown in Example 14. Unfortunately. and the VSWITCH swaps to the other OSA port.624 ms icmp_seq=13 ttl=64 time=0. 64 bytes from 192.168. On a z/VM console.109 192.147/ q vswitch testsw3 <. 8 received.200. Example 13 Ping test across OSA Express port detach lnxsu6:~ # ping -R 192.Tip: Adding the -R switch on the ping command adds the RECORD_ROUTE option to the ICMP echo request packets that ping generates.200. \ <.592/0. Each host along the path adds its address to the IP header.

5. 3. Example 15 TCPIP console log output across switch port deactivation 15:47:52 15:47:52 15:47:52 15:47:52 15:47:52 15:47:52 15:47:52 15:47:52 OSA 2320 OSA 2321 OSA 2322 15:47:52 15:47:52 15:47:52 15:47:52 15:47:52 15:47:52 15:47:52 15:47:52 15:47:52 15:47:52 15:47:57 15:47:57 15:47:57 DTCOSD309W Received adapter-initiated Stop Lan 1 DTCOSD082E VSWITCH-OSD shutting down: 2 DTCPRI385I Device TESTSW32320DEV: DTCPRI386I Type: VSWITCH-OSD.168. QDIO data transfer for the new OSA Express port is activated.01 15:48:11 Active: 1 MFS: 8192 5 VDEV: 2180 MAXCONN: INFINITE ACCOUNTING: OFF QUEUESTORAGE: 8 In the console log of the TCPIP service machine.109 DTCOSD246I VSWITCH-OSD device TESTSW32180DEV: Assigned IPv4 address 192. This output is shown in Example 15.201.VSWITCH SYSTEM TESTSW3 Type: VSWITCH PERSISTENT RESTRICTED NONROUTER State: Ready CONTROLLER: TCPCTL1 IPTIMEOUT: PORTNAME: OSA2180 RDEV: 2180 PORTNAME: OSA2320 RDEV: 2320 Ready. The IP addresses for active guests are registered to the OSA Express port.202. we saw some very good message output. QDIO data transfer on the failed interface is terminated. TCP/IP initializes a new dynamically-defined device for the newly-attached OSA Express port. TCP/IP detaches the real OSA Express devices and attaches the devices for the backup OSA Express port (only the device detachments were logged). 4. The OSA Express notifies TCP/IP that the LAN is no longer available. 7. 2. 30 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.01/0. 6.4 . LAN communications are re-established.168. TCP/IP commences a shutdown of the VSWITCH dynamically-defined device. Detach the OSA from the controller service machine The next thing we tried was to detach the real devices for the VSWITCH’s active OSA Express port from the system.109 DTCOSD246I VSWITCH-OSD device TESTSW32180DEV: Assigned IPv4 address 192.109 7 The chain of events is as follows: 1.200. Status: Not started DTCPRI387I Envelope queue size: 0 DTCPRI388I Address: 2180 DTCQDI001I QDIO device TESTSW32180DEV device number 2182: 6 DTCQDI007I Enabled for QDIO data transfers DTCOSD238I ToOsd: Multicast support enabled for TESTSW32180DEV DTCOSD319I ProcessSetArpCache: Supported for device TESTSW32180DEV DTCOSD341I Obtained MAC address 0006296CA5BC for device TESTSW32180DEV DTCOSD246I VSWITCH-OSD device TESTSW32180DEV: Assigned IPv4 address 192. Status: Ready DTCPRI387I Envelope queue size: 0 DTCPRI388I Address: 2320 DTCQDI001I QDIO device TESTSW32320DEV device number 2322: 3 DTCQDI007I Disable for QDIO data transfers DETACHED TCPCTL1 2320 BY TCPCTL1 4 DETACHED TCPCTL1 2321 BY TCPCTL1 DETACHED TCPCTL1 2322 BY TCPCTL1 DTCOSD080I VSWITCH-OSD initializing: 5 DTCPRI385I Device TESTSW32180DEV: DTCPRI386I Type: VSWITCH-OSD. This allowed us to simulate the OSA Express device becoming unavailable through hardware failure or operator intervention. T=0.168.

time 10006ms rtt min/avg/max/mdev = 0. we issued the commands and received messages as shown in Example 192. take care with your OSA connections. 64 64 64 64 64 bytes bytes bytes bytes bytes from from from from from 192. 6 received.200. \ <. The result of the ping test is shown at Example 16.623 ms icmp_seq=11 ttl=64 time=0.200.100 192.146 ms lnxsu6:~ # You can see that communication was interrupted for a short time while another controller TCPIP stack took control of the VSWITCH.168.641 ms icmp_seq=9 ttl=64 time=0. T=0.200.100) from 192. Failure of the controller service machine We simulated the failure of the controller service machine by using the CP FORCE command on its userid.200.100 ping statistics --11 packets transmitted.200. Almost as soon as the disruption took place. we had to reset the OSA by configuring the channel path offline and back online.100: 192.01 15:48:11 HCPSWU2830I VSWITCH SYSTEM TESTSW3 status is controller not available.200.625 ms icmp_seq=3 ttl=64 time=0.200.100: icmp_seq=1 ttl=64 time=1. 64 bytes from 192.168.100: 192. We found that repeatedly detaching the OSA Express devices from the TCPIP service machines eventually would cause problems with the OSA (channel errors were reported in the TCPIP log when the OSA was used.force here route) route) route) --. On a z/VM console. Example 16 Ping test across controller service machine outage lnxsu6:~ # ping -R 192.621 ms (same (same (same (same (same route) route) <.100 (192. and communications would fail).200.168. Attention: If you want to conduct similar testing.200. Again we used ping to verify the communication path.168.168.100 192.705/1. > unsolicited HCPSWU2830I TCPCTL2 is VSWITCH controller.force here HCPSWU2830I VSWITCH SYSTEM TESTSW3 status is ready.168.100: 192.168. / messages q vswitch testsw3 VSWITCH SYSTEM TESTSW3 Type: VSWITCH Active: 1 MAXCONN: INFINITE PERSISTENT RESTRICTED NONROUTER MFS: 8192 ACCOUNTING: OFF State: Ready CONTROLLER: TCPCTL2 IPTIMEOUT: 5 QUEUESTORAGE: 8 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.100: icmp_seq=2 ttl=64 time=0. 45% loss. Example 17 z/VM commands and messages around controller failure q vswitch testsw3 VSWITCH SYSTEM TESTSW3 Type: VSWITCH Active: 1 MAXCONN: INFINITE PERSISTENT RESTRICTED NONROUTER MFS: 8192 ACCOUNTING: OFF State: Ready CONTROLLER: TCPCTL1 IPTIMEOUT: 5 QUEUESTORAGE: 8 PORTNAME: OSA2180 RDEV: 2180 VDEV: 2180 PORTNAME: OSA2320 RDEV: 2320 Ready.200.697 ms icmp_seq=10 ttl=64 time=0.02 ms RR: 192.168.01/0.109 : 56(124) bytes of data. To recover.200.200.192. the OSA Express detected a loss of communication and the VSWITCH was transferred to the other OSA Express port.200.4 31 .168.100 PING 192.The results were the same as for the switch port deactivation.

0.168.2 has no support for VLANs. we were very satisfied with how the system preserved isolation of the different VLANs used. we attached a guest running Red Hat Linux 7. As distributed.0. VLAN Isolation with untagged frames When setting up and using VLANs in our testing with VSWITCH. Before you rely on this function to protect critical services.2 to our VSWITCH (in Figure 6 on page 28.1 dev hsi0 [root@lnxrh2 root]# 32 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. Ping is only a very simplistic test. scope global eth4 inet scope global eth4 inet 192.10 9. there were two other guests on the VLAN.NOARP. especially with non-VLAN-aware guests.0/24 dev hsi0 scope link 192. both VLAN-aware SLES8 systems.NOARP> mtu 1492 qdisc noop qlen 100 link/ether 02:00:00:00:00:0f brd ff:ff:ff:ff:ff:ff 4: hsi1: <MULTICAST. When we granted access to the VSWITCH.9.168. We then tried configuring different IP addresses as secondary addresses on the eth4 interface of the guest (the interface that connected to the VSWITCH).01 16:47:38 Important: We are very impressed with the way that z/VM Virtual Switch provides failover for its components.100.0/24 via 192.12. Red Hat 7.NOARP.UP> mtu 1492 qdisc pfifo_fast qlen 100 link/ether 02:00:00:00:00:10 brd ff:ff:ff:ff:ff:ff inet 192.168.UP> mtu 16384 qdisc pfifo_fast qlen 100 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff inet conduct your own testing to verify that the protocols and applications that you run in your environment recover from the brief outage interval. and verified that we could communicate between it and other guests on the VLAN.0/24 dev eth4 proto kernel scope link src 192.100. To test this.9.01/0.12.201. with only a short interval of dead time. we granted it only to VLAN 201. T=0. this was the guest LNXRH2).10/24 scope global eth4 3: eth2: <MULTICAST.UP> mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0/8 dev lo scope link default via 9.168.255 scope global hsi0 [root@lnxrh2 root]# ip route ls 192.NOARP> mtu 8192 qdisc noop qlen 100 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff 5: hsi0: <MULTICAST.12.255 scope host lo 2: eth4: <MULTICAST.0/24 dev eth4 proto kernel scope link src 192.168.PORTNAME: OSA2180 RDEV: 2180 VDEV: 2180 PORTNAME: OSA2320 RDEV: 2320 Ready. brd 192.201. We configured the Red Hat guest with an IP address configuration for that VLAN.9. Example 18 IP configuration on LNXRH2 [root@lnxrh2 root]# ip addr ls 1: lo: <LOOPBACK.10 127.168. Other than our Red Hat guest.0/24 dev eth4 scope link brd 127.200.0 dev eth4 192.202.45/24 brd 9. the connectivity between the VSWITCH and the LAN is restored. We wanted to verify what happened when untagged frames were used.4 .202.168.

10 : 56(84) bytes of data. network.168. 3 packets received.255. You can inspect this table using the CP QUERY VSWITCH command with the DETAIL option. we were surprised to find that we did get responses. (192.100.10 Mask: 255. The VSWITCH uses this table to determine which guest to send a received packet to.308/0.0. Example 20 CP QUERY VSWITCH for IP address table q vswitch testsw3 detail VSWITCH SYSTEM TESTSW3 Type: VSWITCH Active: 3 MAXCONN: INFINITE PERSISTENT RESTRICTED NONROUTER MFS: 8192 ACCOUNTING: OFF State: Ready CONTROLLER: TCPCTL1 IPTIMEOUT: 5 QUEUESTORAGE: 8 PORTNAME: OSA2320 RDEV: 2320 VDEV: 2320 PORTNAME: OSA2180 RDEV: 2180 RX Packets: 30569 Discarded: 33 Errors: 0 TX Packets: 27494 Discarded: 0 Errors: 0 RX Bytes: 2744464 TX Bytes: 2305386 Authorized userids: LNXRH2 VLAN: 0201 LNXSU6 VLAN: ANY LNXSU7 VLAN: 0201 0202 SYSTEM VLAN: ANY VSWITCH Connection: Device: 2322 Unit: 002 Role: DATA VLAN: ANY Assigned by user Unicast IP Addresses: 192. Example 19 Ping command on LNXRH2 [root@lnxrh2 root]# ping 192.109 ping statistics --3 packets transmitted.0.359/0.202. we had to look closer into the action of both the VSWITCH code and the Linux kernel.168. By configuring IP addresses belonging to other VLANs against our interface.0/24 network.168.0/24 network.201. we tell the kernel to try and reach those networks directly instead of routing. We should only be able to communicate directly with other guests on that same VLAN.100. When we tried to ping the hosts on the 192.109 PING 192.041 ms [root@lnxrh2 root]# To try and understand what was happening here.100.168.202 Mask: 0.10 Mask: 255.109: icmp_seq=0 ttl=64 time=359 usec 64 bytes from 192. we got no responses.109) from 192.255.168. 64 bytes from 192. corresponding to the 192.168. Example 20 shows this command output after the ping command above.100.255.0 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4.0 192.Our Red Hat guest was authorized only to VLAN 201.109: icmp_seq=2 ttl=64 time=275 usec --. when we then tried to ping a host on the 192. This confirmed that we were being isolated from VLAN 202.255.109: icmp_seq=1 ttl=64 time=292 usec 64 bytes from 192. However.100.100.0 192.4 33 .168. 0% packet loss round-trip min/avg/max/mdev = 0.0 Remote Adapter Owner: LNXRH2 NIC: 6100 Name: TESTSW3 Device: 6102 Unit: 002 Role: DATA VLAN: 0201 Assigned by system Unicast IP Addresses: 192.255.255. The VSWITCH maintains a table of connected guests and associated IP addresses. Mask: 255.275/ even though we had restricted the guest to VLAN 201.

4.0. The IP stack in LNXRH2 checks to see if the destination IP address is local.255. so the next check is whether the VLAN authority of LNXSU6 will allow the frame to be delivered. 1.01/0.4 . Because LNXSU6 is authorized to VLAN ANY.168.202. since the two machines were configured with a common network interface and were able to communicate anyway.0.168. The source address is 192. LNXSU6 has registered this address. Untagged frames are not filtered by VSWITCH.0.1 MAC: 01-00-5E-00-00-01 Adapter Owner: LNXSU6 NIC: 6100 Name: TESTSW3 Device: 6102 Unit: 002 Role: DATA VLAN: ANY Assigned by user Unicast IP Addresses: 192. this time for the destination address 192. VSWITCH takes no action on the untagged frame entering the VLAN. Next. formatting an ICMP echo reply to be sent to 192. and because LNXRH2 is authorized to only one VLAN ID the VSWITCH tags the frame for VLAN 201. 34 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. work this way to improve performance. so the packet is given to its destination (the ping program). It checks the IP address table again. Rather than forwarding the packet Many IP stacks through the stack to deliver it at the correct interface. the VSWITCH looks up the IP address table for the destination address. The main reason this connection can take place is an efficiency aspect of the Linux kernel. The IP stack in LNXSU6 does a final check to see if the destination address is local or if the packet has to be routed. T=0.01 15:22:41 The following chain of events takes place during our ping.168.100. so the packet handled by the kernel. The other reason that the connection works is that the VSWITCH does not filter untagged frames for delivery to guests.1 MAC: 01-00-5E-00-00-01 FF02::1 MAC: 33-33-00-00-00-01 Local FF02::1:FF00:8 MAC: 33-33-FF-00-00-08 Local Ready.10. which it is in this case. so the frame is sent to LNXRH2.0 192. 6. and finds it on LNXRH2.100.109 Mask: 255. it technically has an incorrect destination address for the interface it is arriving on. when the packet arrives at the VLAN interface. The route table selects 192. so VSWITCH delivers the frame. This packet will be sent as an untagged frame. 5. the kernel will simply accept it at the interface it was delivered.Multicast IP Addresses: 224.109 Mask: the fact this happens this way is a curiosity. At step 3.100.0 FE80::200:0:300:8/0 Local Multicast IP Addresses: 224. and sends the reply via the eth4 interface. LNXRH2 sends an ICMP echo request packet. However if the machines were not intended to have a shared network then this could lead to unexpected connectivity and a loss of security. because we have that address configured on one of our local interfaces and the route table selected it.109 Mask: 255.168. 2. The kernel processes the ICMP echo request.109 as the source address.255.255.0 192. The address is local. The packet is given to the VSWITCH. LNXSU6 is authorized to VLAN ANY. In this instance.

the base interface (eth4) was used to gain connectivity to the other Linux guests that were attached to the other OSA device in the network (the 192. as far as the switch was concerned. you can prevent the kernel from accepting packets on an interface whose network does not match the destination address of the packet. we are drawing your attention to an interoperation aspect that you should be aware of when implementing VSWITCH and VLAN in a Linux environment. this configuration eliminated untagged frames from LNXSU6. Eliminate untagged frames In the configuration of LNXSU6. Attention: To use tcpdump with QETH network interfaces. By adding iptables rules in any Linux host attached to multiple VLANs. we discuss these methods in more detail. a VLAN that LNXRH2 was not authorized to. Rather. we could still communicate with the other machines on the 192. Firewall rules in Linux Many administrators do not like the default behavior of Linux accepting packets on “foreign” interfaces. Importantly. so all the Linux guests attached to the other OSA were actually in VLAN 1. By changing the configuration of LNXSU6 to bring up a connection to the 192.168. or by using firewall rules in Linux. Analyzing the captured packets.Closing the door There are at least two ways you can eliminate this possible exposure: by eliminating untagged frames.168.100. The Cisco switch that we connected to was defined to use VLAN 1 as a default VLAN ID.0/24 network via VLAN 1 and using a dummy IP address on the base interface. We started a packet capture against the VLAN 2 interface (the 192. and issued ping commands for destinations on the various VLANs that our host was attached to.100. Attention: We are not trying to point out a design problem by illustrating this issue. and the Ethereal analysis tool to look at the trace. we do not believe that a design problem exists.168. This was what we expected. Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. Figure 7 on page 36 is a screen capture of the Ethereal program analyzing our captured traffic. This script is included in the tcpdump package on SuSE SLES 8.0/24 network).100.0/24 network. Capturing VLAN network traffic on VSWITCH We used the tcpdump packet capture tool to grab network traffic.168. we only saw the packets on VLAN2.4 35 . The frames sent back to LNXRH2 were not delivered by the VSWITCH because they were tagged as VLAN 1.0/24 network).200. In the following sections. the tcpdump-qeth wrapper script by Holger Smolinski of IBM is needed to analyze the frames correctly.

This is the dummy Ethernet header inserted into the frame by the tcpdump-qeth script.109 Mask:255. Example 21 Interface statistics for a VLAN interface lnxsu6:~ # ifconfig vlan2 vlan2 Link encap:Ethernet HWaddr 02:00:00:00:00:08 inet addr:192. it would seem that the frames are being delivered to the base interface and accepted there rather than registering against the VLAN interface. In this case. you can see that the Ethernet II header has all-zero MAC addresses for source and destination MAC.0 inet6 addr: fe80::200:0:300:8/10 Scope:Link 36 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. Note: In the Ethereal display shown in Figure 7. Looking at the statistics for the VLAN 2 interface.Figure 7 Ethereal packet analysis on VLAN 2 We highlighted one packet for display in the decode window. which is Linux receiving packets on any configured interface as long as the IP address is local. this definitely appears to be the case (refer to Example 21).255. It appears that the same “feature” that we discussed in “VLAN Isolation with untagged frames” on page 32.4 .255. You can also see that we only appear to have captured the transmitted packets.200.168. is apparent here.

Seeing this behavior. Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. Two things surprised us about what we saw in the trace of the base interface: We saw no VLAN headers We saw both transmitted and received IP packets for all VLANs Figure 8 on page 38 shows Ethereal analyzing a packet trace we collected against the eth4 interface on LNXSU6. 2 Promiscuous mode gave the same result. which shows it is fairly meaningless on z/VM simulated networks anyway. In doing so we hoped to see frames with VLAN headers in the trace.0 b) TX bytes:664403 (648. which prevents tcpdump from using promiscuous mode (we wanted to be absolutely sure we were not opening anything up that we did not intend to2). we then traced the base interface.UP RUNNING MULTICAST MTU:1492 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:7783 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0. We wanted to see the contents of the VLAN headers and other changes to packets that the VLAN support made. we ran tcpdump with the -p option.4 37 .8 Kb) You can see that no received packets have been recorded against this interface. In particular.

this still may not produce the desired results because the Ethernet header is not always preserved by the OSA Express hardware. the QETH driver has been released as open source. for example. not all frames received on a QETH interface will ever have had an Ethernet header: traffic between guests on the same simulated network. By the time tcpdump views the packets. this makes sense. Since QETH is an IP-only transport. The source and destination addresses for these packets are for addresses on VLAN 2 and VLAN 202 in our configuration.21 kernel. However. you can see some of the packets that were collected. do not. Note: With the IBM s390 kernel patches for the 2. An interesting project would be to modify the driver so that “traditional” tools like libpcap function as expected. 38 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. Also.1Q processing has finished with them. but of other frame headers as well. the 802. This may allow better visibility not only of VLAN frames.Figure 8 Ethereal packet analysis on base interface In the top portion of the window. It appears that tcpdump can only see IP frames when QETH is used.4 . having extracted the VLAN headers and leaving just the IP payload.4.

| | V (2) | | |-RDEV--.4 39 . as long as switchname is the first operand specified.-------->< | | v--------.-VLAN--ANY--------.-GRAnt--userid--+------------------+-.-QUEuestorage--8M------. | | V (3)| | '-PORTname----portname----+-' Notes: (1) You can specify the operands in any order. >>--SET VSWITCH--switchname--. DEFINE VSWITCH The syntax of the DEFINE VSWITCH command is shown in Example 22. .-------------. Example 22 DEFINE VSWITCH syntax DEFINE VSWITCH (1) . . >--+-----------------------+--+---------------------+--+----------------+----> '-QUEuestorage--numberM-' '-CONTRoller--userid1-' '-IPTimeout--nnn-' .---------.--------------. You can specify a maximum of 3 port names. | | | '-VLAN----vlanid-+-' | |-REVoke--userid----------------------| | .-CONTRoller--*-------.-CONnect----.Command usage The usage syntax of the commands used in this paper are shown in this section.---------------| | '-NONE--------' | Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. (2) (3) You can specify a maximum of 3 real device numbers. Refer to z/VM CP Command and Utility Reference for full details of all parameters. and portname is the last operand specified.-NONrouter-. >--+-----------+--. . if applicable.----------.---------------------------. | | V (1) | | |-PORTname----portname-----+----------| | .---rdev-----+. SET VSWITCH The syntax of the SET VSWITCH command is shown in Example 23.-RDEV--NONE--------. Example 23 SET VSWITCH syntax SET VSWITCH .-IPTimeout--5---.----------------------------->< '-PRIrouter-' | . | '-DISCONnect-' | V (2)| | '-RDEV----rdev----+-' . >>--DEFINE VSWITCH--switchname------+-------------------+--+------------+----> | .

--NICDEF--vdev---------------------------> | (1)| '-Prefix Keywords----' >--.--lanname-. DIRMAINT NICDEF Example 24 shows the options available on the DIRMAINT NICDEF command.-DEVices--devs-----------------.-DELETE-----------------------------------------.-' '-QDIO---------' '-| NICDEF Options |-' NICDEF Options: v----------------------------------.-+--------------------------------------| |-LAN--.4 . enter the following command: dirm help dirmaint vconfig Example 25 shows the usage of the vconfig command. Refer to z/VM CP Command and Utility Reference for full details of all parameters. inclusive.-HIPERsockets-. Usage: add rem Was: 1 [interface-name] [vlan_id] [vlan-name] 40 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. Example 25 Usage of the vconfig command lnxsu6:~ # vconfig Expecting argc to be 3-5.-ownerid-.-.--------------------.----------------------->< '-TYPE--.|-CONnect-----------------------------| |-DISCONnect--------------------------| |-QUEuestorage--numberM---------------| |-CONTRoller--. |---.--.-*-------.--------------------. (2) You can specify a maximum of 3 real device numbers.-| | | |-*-------| | | | | '-SYSTEM--' | | | '-SYSTEM--switchnm-----' | |-CHPID--hh---------------------| '-MACID--macaddress-------------' Note: (1) For more information on prefix keywords.-------------| | '-userid1-' | |-IPTimeout--nnn----------------------| |-NONrouter---------------------------| '-PRIrouter---------------------------' Notes: (1) You can specify a maximum of 3 port names. Example 24 DIRMAINT NICDEF command NICDEF >>--DIRMaint--.

Poughkeepsie Center.set_flag set_egress_map set_ingress_map set_name_type [interface-name] [flag-num] [vlan-name] [skb_priority] [vlan-name] [skb_priority] [name-type] [0 | 1] [vlan_qos] [vlan_qos] * The [interface-name] is the name of the ethernet card that hosts the VLAN you are talking about. the VLAN device will move the ethernet header around to make it look exactly like a real ethernet device. eight of which have been spent working on S/390 and zSeries. DEV_PLUS_VID (eth0. SG24-6299. If you don't need it. REDP3657. He holds a Bachelor of Computing Science degree from Queensland University of Technology. Vic Cross is the Linux for zSeries and S/390® Team Leader at Independent Systems Integrators. * skb_priority is the priority in the socket buffer (sk_buff).5) * bind-type: PER_DEVICE # Allows vlan 5 on eth0 and eth1 to be unique. PER_KERNEL # Forces vlan 5 to be unique across all devices. Linux on IBM ^ zSeries and S/390: Porting LEAF to Linux on zSeries. VLAN_PLUS_VID_NO_PAD (vlan5). He has more than 15 years of experience in general computing.4 41 . This may help programs such as DHCPd which read the raw ethernet packet and make assumptions about the location of bytes. Poughkeepsie Center Angelo Macchiano and Dennis Musselwhite z/VM Development. The author of this Redpaper This Redpaper was produced by a team of specialists from around the world working at the International Technical Support Organization. because there will be at least a small performance degradation.0005). REDP3627. * FLAGS: 1 REORDER_HDR When this is set. DEV_PLUS_VID_NO_PAD (eth0. He is a co-author of IBM Redbooks™ and Redpapers Linux on IBM ^ zSeries and S/390: ISP/ASP Solutions. Default is OFF. His areas of expertise include networking and Linux. * vlan_qos is the 3 bit priority in the VLAN header * name-type: VLAN_PLUS_VID (vlan0005). and Linux on IBM ^ zSeries and S/390: Virtual Router Redundancy Protocol on VM Guest LANs. Linux on IBM ^ zSeries and S/390: Large Scale Linux Deployment. Thanks to the following people for their contributions to this project: Roy Costa and Greg Geiselhart International Technical Support Organization. * The vlan_id is the identifier (0-4095) of the VLAN you are operating on. don't turn it on. IBM’s Large Systems Business Partner in Australia. IBM Endicott Simon Williams IBM Australia Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. SG24-6824.

SC24-6008 42 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. SC24-5955 z/VM CP Command and Utility Reference.4 .Related publications The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this Redpaper. IBM Redbooks Linux on IBM ^ zSeries and S/390: Virtual Router Redundancy Protocol on VM Guest LANs. REDP3657 Other publications z/VM Virtual Machine Operation.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. To illustrate them as completely as possible. This information contains examples of data and reports used in daily business operations.A. or features discussed in this document in other countries. EITHER EXPRESS OR IMPLIED. and products. All rights reserved. The furnishing of this document does not give you any license to these patents. program. marketing. or service. You may copy. using. cannot guarantee or imply reliability.S. services. this statement may not apply to you. IBM.A. You may copy. © Copyright IBM Corp. program. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND. These examples have not been thoroughly tested under all conditions. modify. However. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. the examples include the names of individuals. these changes will be incorporated in new editions of the publication. and distribute these sample programs in any form without payment to IBM for the purposes of developing. serviceability. You can send license inquiries. their published announcements or other publicly available sources. it is the user's responsibility to evaluate and verify the operation of any non-IBM product.Notices This information was developed for products and services offered in the U. in writing. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Changes are periodically made to the information herein. IBM Corporation. which illustrates programming techniques on various operating platforms. North Castle Drive Armonk. THE IMPLIED WARRANTIES OF NON-INFRINGEMENT. or service may be used.S. IBM has not tested those products and cannot confirm the accuracy of performance. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. to: IBM Director of Licensing. compatibility or any other claims related to non-IBM products. Some states do not allow disclaimer of express or implied warranties in certain transactions. or service is not intended to state or imply that only that IBM product. COPYRIGHT LICENSE: This information contains sample application programs in source language. and distribute these sample programs in any form without payment to IBM. therefore. BUT NOT LIMITED TO. NY 10504-1785 U. 43 . program. or distributing application programs conforming to IBM's application programming interfaces. or service that does not infringe any IBM intellectual property right may be used instead. This information could include technical inaccuracies or typographical errors. for the purposes of developing. program. companies. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. brands. or function of these programs. IBM may have patents or pending patent applications covering subject matter described in this document. INCLUDING. 2003. Any functionally equivalent product. Any reference to an IBM product. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. Information concerning non-IBM products was obtained from the suppliers of those products. IBM may not offer the products. modify. Consult your local IBM representative for information on the products and services currently available in your area. marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. therefore. using.

and Pentium are trademarks of Intel Corporation in the United States. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems. Windows.This document created or updated on September 3. Other company. in the United States. Intel Inside (logos). other countries. and service names may be trademarks or service marks of others. 2003.A. MMX. 44 Linux on IBM ^ zSeries and S/390: VSWITCH and VLAN Features of z/VM 4. or both. International Technical Support Organization Dept. ® Trademarks The following terms are trademarks of the International Business Machines Corporation in the United States. UNIX is a registered trademark of The Open Group in the United States and other countries. SET. other countries. SET Secure Electronic Transaction. HYJ Mail Station P099 2455 South Road Poughkeepsie.4 . or both: Domino™ HiperSockets™ ibm. Inc. Send us your comments in one of the following ways: Use the online Contact us review redbook form found at: Send your comments in an Internet note to: redbook@us. or other countries. other countries. Microsoft. and the SET Logo are trademarks owned by SET Secure Electronic Transaction Mail your comments to: IBM Corporation. or both. Windows NT. and the Windows logo are trademarks of Microsoft Corporation in the United States.® IBM® Lotus® MVS™ Notes® Redbooks(logo) Redbooks™ S/390® ™ ™ z/OS® z/VM® zSeries® The following terms are trademarks of other companies: Intel. NY 12601-5400 U.