Ethtool Support for Microsoft Hyper-V

From Mindworks
Jump to: navigation, search

Ethtool Support for Microsoft Hyper-V is a programming project to implement support of the Linux command-line application "ethtool" on virtual machine guests of the Microsoft Hyper-V hypervisor. Ethtool is used to query and control network driver and hardware settings. Enabling support requires patching the virtual network driver in the Linux kernel.

Hyper-Tux Logo
Sponsors Microsoft Linux Integration Service
Mentors
  • K.Y. Srinivasan
  • Haiyang Zhang
Team Name Hyper-Tux
Duration Fall 2014 - Spring 2015


Students
  • Andrew Schwartzmeyer
  • Justin Hall
  • Keith Drew
  • Eric Doman
  • Chris Waltrip (Fall 2014)

Problem Statement

Ethtool[1] is a command-line utility for Linux-kernel based operated systems, maintained in parallel with the kernel. Ethtool is primarily used by system administrators to query and/or set network driver variables such as link status, number of send and receive queues of vRSS, size of send and receive buffers, available features, and gathered statistics.

The network driver used by Linux virtual machines running on Microsoft Hyper-V[2] minimally supports ethtool; it can return the name of the driver, "hv_netvsc," and the permanent address of the network device. All Linux-supported network drivers are responsible for implementing the ethtool API. The goal of this project is to implement support for all common uses of ethtool.

Background

A common scenario for system administrators is to host multiple virtual machines[3] on the same piece of hardware, so one physical computer and its host operating system can support many guest computers, each with its own operating system. This process of "virtualization" [4] is dependent on an abstraction layer called a hypervisor[5]. Microsoft's solution is a native hypervisor, Microsoft Hyper-V, included in most versions of Microsoft Windows. A native hypervisor runs directly on the host's hardware, in contrast to a hosted hypervisor which runs on the host's operating system, so the former has fewer layers of translation for guest operating system instructions.

Specifications

Since the guest operating systems are separate from the host operating system, a host machine running Windows with Hyper-V supports guests[6] running Linux (as well as Windows). Linux supports being run as a guest operating system on a hypervisor through virtual drivers for the guest's virtual hardware, including video, storage, keyboard, mouse and networking. These drivers are a part of the Linux kernel, and are maintained by our client, the Microsoft Linux Integration Services team. We are concerned with the virtual network driver, netvsc[7].

Design

Ethtool is a mature application and its programming[8] interface[9] in the kernel is well-established. In general, adding support for each ethtool operation requires patching netvsc[10] with the appropriate new callback for the operation and any additional necessary changes.

Goals

Our current goal is to implement support for the following ethtool operations:

Table 1: Implementation Goals
Operation Description Status
show-channels Queries the network device for the numbers of channels it has. A channel is an IRQ and the set of queues that can trigger that IRQ. Complete
set-channels Changes the numbers of channels of the specified network device. Testing
show-features Queries the specified network device for the state of protocol offload and other features. Stretch Goal
show-ring Queries the specified network device for rx/tx ring parameter information. Stretch Goal
set-ring Changes the rx/tx ring parameters of the specified network device. Stretch Goal
statistics Queries the specified network device for network interface card and driver specific statistics. Stretch Goal

Implementation

Our work in progress is tracked on our issues page[11].

Currently, we have implemented the show-channels[12] operation. Our patch was signed-off[13] by Haiyang Zhang, and applied[14] to the net-next kernel tree by Dave Miller. The patch was commited[15] and is now in the development version of the Linux kernel.

Our next goal is to implement the set-channels operation. We several issues to finalize before our final patch submission. These issues include:

Table 2: Current Issues
Major Requirements Description Operation Status
work syncing[16] Determine the proper way to pause and re-sync the work over the new channels. set-channel Closed
NVSP version[17] Determine if every NVSP version can support changing num_chn. set-channel Closed
netif_set_real_num_tx_queues()[18] Develop error checking since netif_set_real_num_tx_queues() can fail. set-channel Closed
Host-side Settings[19] Check if host supports RSS, Intel x520 etc. Ensure VMQ is enabled. set-channel Closed

Testing

Testing patches to a virtual network driver on a Linux machine is a tedious process. It requires direct access to a host machine running Windows with Hyper-V, and a Linux virtual machine for development and testing. When testing the network driver your machine is dependent on, if your patch causes the driver to break, direct access through the console is needed to debug, as network access will be disabled.

Additionally, developing Linux kernel patches requires configuring and installing the correct development version of the Linux kernel to the test machine. For networking drivers, we work off Dave Miller's net-next tree[20]. Our team primarily develops on CentOS 7 virtual machines, since installing a locally compiled kernel is made relatively easy with the kernel build system's creation of a native package.

We wrote a Linux development compilation reference guide[21] for each time test machine we setup.

Our normal workflow is to develop on a local clone of the kernel tree, and then push the patch to our test machine, where we can compile and install the patched netvsc module. When our patch breaks, we use the Remote Desktop Protocol to connect to the host Windows machine, to then use the Hyper-V manager's to connect to the Linux virtual machine without requiring the guest's networking to be operational. Further debugging is done through liberal use of printk, the kernel's version of printf for printing out debug information.

Testing the patches under load requires an extensive hardware setup: a Windows Server 2012 R2 machine running Hyper-V, with a 1 Gbps network adapter which supports Receive Side Scaling, so that the virtual machines can use virtual RSS. This resource is currently being provisioned.

Project Deliverables

get-channels
  • From: Andrew Schwartzmeyer <andrew@schwartzmeyer.com>
  • Date: Wed, 25 Feb 2015 19:49:24 -0800
  • Subject: [PATCH net-next] hyperv: Implement netvsc_get_channels() ethool op

This adds support for reporting the actual and maximum combined channels count of the hv_netvsc driver via 'ethtool --show-channels'.

This required adding 'max_chn' to 'struct netvsc_device', and assigning it 'rsscap.num_recv_que' in 'rndis_filter_device_add'. Now we can access the combined maximum channel count via 'struct netvsc_device' in the ethtool callback.

Signed-off-by: Andrew Schwartzmeyer <andrew@schwartzmeyer.com>

---
 drivers/net/hyperv/hyperv_net.h   |  1 +
 drivers/net/hyperv/netvsc_drv.c   | 14 ++++++++++++++
 drivers/net/hyperv/rndis_filter.c |  6 +++++-
 3 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/drivers/net/hyperv/hyperv_net.h b/drivers/net/hyperv/hyperv_net.h
index 384ca4f4de4a..4815843a6019 100644
--- a/drivers/net/hyperv/hyperv_net.h
+++ b/drivers/net/hyperv/hyperv_net.h
@@ -634,6 +634,7 @@ struct netvsc_device {
 
 	struct vmbus_channel *chn_table[NR_CPUS];
 	u32 send_table[VRSS_SEND_TAB_SIZE];
+	u32 max_chn;
 	u32 num_chn;
 	atomic_t queue_sends[NR_CPUS];
 
diff --git a/drivers/net/hyperv/netvsc_drv.c b/drivers/net/hyperv/netvsc_drv.c
index 15d82eda0baf..a06bd6614007 100644
--- a/drivers/net/hyperv/netvsc_drv.c
+++ b/drivers/net/hyperv/netvsc_drv.c
@@ -687,6 +687,19 @@ static void netvsc_get_drvinfo(struct net_device *net,
 	strlcpy(info->fw_version, "N/A", sizeof(info->fw_version));
 }
 
+static void netvsc_get_channels(struct net_device *net,
+				struct ethtool_channels *channel)
+{
+	struct net_device_context *net_device_ctx = netdev_priv(net);
+	struct hv_device *dev = net_device_ctx->device_ctx;
+	struct netvsc_device *nvdev = hv_get_drvdata(dev);
+
+	if (nvdev) {
+		channel->max_combined	= nvdev->max_chn;
+		channel->combined_count = nvdev->num_chn;
+	}
+}
+
 static int netvsc_change_mtu(struct net_device *ndev, int mtu)
 {
 	struct net_device_context *ndevctx = netdev_priv(ndev);
@@ -760,6 +773,7 @@ static void netvsc_poll_controller(struct net_device *net)
 static const struct ethtool_ops ethtool_ops = {
 	.get_drvinfo	= netvsc_get_drvinfo,
 	.get_link	= ethtool_op_get_link,
+	.get_channels   = netvsc_get_channels,
 };
 
 static const struct net_device_ops device_ops = {
diff --git a/drivers/net/hyperv/rndis_filter.c b/drivers/net/hyperv/rndis_filter.c
index 7816d98bdddc..ca81de04bc76 100644
--- a/drivers/net/hyperv/rndis_filter.c
+++ b/drivers/net/hyperv/rndis_filter.c
@@ -1027,6 +1027,7 @@ int rndis_filter_device_add(struct hv_device *dev,
 
 	/* Initialize the rndis device */
 	net_device = hv_get_drvdata(dev);
+	net_device->max_chn = 1;
 	net_device->num_chn = 1;
 
 	net_device->extension = rndis_device;
@@ -1094,6 +1095,7 @@ int rndis_filter_device_add(struct hv_device *dev,
 	if (ret || rsscap.num_recv_que < 2)
 		goto out;
 
+	net_device->max_chn = rsscap.num_recv_que;
 	net_device->num_chn = (num_online_cpus() < rsscap.num_recv_que) ?
 			       num_online_cpus() : rsscap.num_recv_que;
 	if (net_device->num_chn == 1)
@@ -1140,8 +1142,10 @@ int rndis_filter_device_add(struct hv_device *dev,
 	ret = rndis_filter_set_rss_param(rndis_device, net_device->num_chn);
 
 out:
-	if (ret)
+	if (ret) {
+		net_device->max_chn = 1;
 		net_device->num_chn = 1;
+	}
 	return 0; /* return 0 because primary channel can be used alone */
 
 err_dev_remv:
-- 
2.3.0
set-channels

From aecd06e665323c07aae18ca9f394a4de282a12c3 Mon Sep 17 00:00:00 2001 From: Andrew Schwartzmeyer <andrew@schwartzmeyer.com> Date: Mon, 27 Apr 2015 22:45:18 -0700 Subject: [PATCH net-next] hyperv: Implement netvsc_set_channels() ethtool op

  • Remove and add RNDIS device with new channel count
  • Added num_chn to struct device_info so it can be passed to
 rndis_filter_device_add()
  • Initialize device_info to 0 always
  • Set num_chan in rndis_filter_device_add() when passed via
 struct device_info, else use num_cpus()
  • Add local u32 num_chan to rndis_filter_device_add() for shorter name
  • Declare netvsc_link_change
  • Check that NVSP version is at least 5
  • Check given combined count against bounds [1, num_online_cpus()]
  • Use num_online_cpus() if given count is too high (and log a notice)
  • Check return values of netif set tx/rx queues
  • Attempt to recover RNDIS if it fails with new setting

Signed-off-by: Andrew Schwartzmeyer <andrew@schwartzmeyer.com>

---
 drivers/net/hyperv/hyperv_net.h   |   1 +
 drivers/net/hyperv/netvsc_drv.c   | 126 ++++++++++++++++++++++++++++++++++++++
 drivers/net/hyperv/rndis_filter.c |  18 ++++--
 3 files changed, 140 insertions(+), 5 deletions(-)

diff --git a/drivers/net/hyperv/hyperv_net.h b/drivers/net/hyperv/hyperv_net.h
index a10b31664709..318f1f0fa1e6 100644
--- a/drivers/net/hyperv/hyperv_net.h
+++ b/drivers/net/hyperv/hyperv_net.h
@@ -162,6 +162,7 @@ struct netvsc_device_info {
 	unsigned char mac_adr[ETH_ALEN];
 	bool link_state;	/* 0 - link up, 1 - link down */
 	int  ring_size;
+	u32  num_chn;
 };
 
 enum rndis_device_state {
diff --git a/drivers/net/hyperv/netvsc_drv.c b/drivers/net/hyperv/netvsc_drv.c
index a3a9d3898a6e..cfe6e8d59c3c 100644
--- a/drivers/net/hyperv/netvsc_drv.c
+++ b/drivers/net/hyperv/netvsc_drv.c
@@ -52,6 +52,8 @@ static int ring_size = 128;
 module_param(ring_size, int, S_IRUGO);
 MODULE_PARM_DESC(ring_size, "Ring buffer size (# of pages)");
 
+static void netvsc_link_change(struct work_struct *);
+
 static void do_set_multicast(struct work_struct *w)
 {
 	struct net_device_context *ndevctx =
@@ -729,12 +731,130 @@ static void netvsc_get_channels(struct net_device *net,
 	}
 }
 
+static int netvsc_set_channels(struct net_device *net,
+			       struct ethtool_channels *channel)
+{
+	struct net_device_context *net_device_ctx = netdev_priv(net);
+	struct hv_device *dev = net_device_ctx->device_ctx;
+	struct netvsc_device *nvdev = hv_get_drvdata(dev);
+	struct netvsc_device_info device_info;
+	const int num_cpus = num_online_cpus();
+	const int num_chn = nvdev->num_chn;
+	int ret;
+
+	if (!nvdev || nvdev->destroy)
+		return -ENODEV;
+
+	if (!channel) {
+		pr_warn("struct ethtool_channels was null\n");
+		return -EINVAL;
+	}
+
+	if (nvdev->nvsp_version < NVSP_PROTOCOL_VERSION_5) {
+		pr_info("vRSS unsupported before NVSP Version 5\n");
+		return -EINVAL;
+	}
+
+	if (channel->combined_count < 1)
+		return -EINVAL;
+
+	/* use max if combined count exceeds it */
+	if (channel->combined_count > num_cpus) {
+		pr_info("set channels too high, using %d\n", num_cpus);
+		channel->combined_count = num_cpus;
+	}
+
+	/* remove RNDIS */
+	nvdev->start_remove = true;
+	cancel_work_sync(&net_device_ctx->work);
+
+	netif_tx_disable(net);
+	rndis_filter_device_remove(dev);
+
+	/* set new num_chn */
+	nvdev->num_chn = channel->combined_count;
+
+	/* add RNDIS */
+	net_device_ctx->device_ctx = dev;
+	hv_set_drvdata(dev, net);
+
+	memset(&device_info, 0, sizeof(device_info));
+	device_info.num_chn = nvdev->num_chn;
+	device_info.ring_size = ring_size;
+
+	ret = rndis_filter_device_add(dev, &device_info);
+	if (ret != 0)
+		goto recover;
+
+	ret = netif_set_real_num_tx_queues(net, nvdev->num_chn);
+	if (ret != 0)
+		goto recover;
+
+	ret = netif_set_real_num_rx_queues(net, nvdev->num_chn);
+	if (ret != 0)
+		goto recover;
+
+	netif_tx_wake_all_queues(net);
+
+	return 0;
+
+ recover:
+
+	/* remove RNDIS */
+	nvdev->start_remove = true;
+	cancel_work_sync(&net_device_ctx->work);
+
+	netif_tx_disable(net);
+	rndis_filter_device_remove(dev);
+
+	/* set previous num_chn */
+	nvdev->num_chn = num_chn;
+
+	/* add RNDIS */
+	net_device_ctx->device_ctx = dev;
+	hv_set_drvdata(dev, net);
+
+	memset(&device_info, 0, sizeof(device_info));
+	device_info.num_chn = nvdev->num_chn;
+	device_info.ring_size = ring_size;
+
+	ret = rndis_filter_device_add(dev, &device_info);
+	if (ret != 0) {
+		netdev_err(net, "unable to add netvsc device (ret %d)\n", ret);
+		goto error;
+	}
+
+	ret = netif_set_real_num_tx_queues(net, nvdev->num_chn);
+	if (ret != 0) {
+		netdev_err(net, "could not set tx queue count (ret %d)\n", ret);
+		goto error;
+	}
+
+	ret = netif_set_real_num_rx_queues(net, nvdev->num_chn);
+	if (ret != 0) {
+		netdev_err(net, "could not set rx queue count (ret %d)\n", ret);
+		goto error;
+	}
+
+	netif_tx_wake_all_queues(net);
+
+	return 0;
+
+ error:
+	free_netdev(net);
+	hv_set_drvdata(dev, NULL);
+	netdev_err(net, "net device not enabled (ret %d)\n", ret);
+
+	return ret;
+}
+
 static int netvsc_change_mtu(struct net_device *ndev, int mtu)
 {
 	struct net_device_context *ndevctx = netdev_priv(ndev);
 	struct hv_device *hdev =  ndevctx->device_ctx;
 	struct netvsc_device *nvdev = hv_get_drvdata(hdev);
 	struct netvsc_device_info device_info;
+
 	int limit = ETH_DATA_LEN;
 
 	if (nvdev == NULL || nvdev->destroy)
@@ -756,7 +876,10 @@ static int netvsc_change_mtu(struct net_device *ndev, int mtu)
 
 	ndevctx->device_ctx = hdev;
 	hv_set_drvdata(hdev, ndev);
+
+	memset(&device_info, 0, sizeof(device_info));
 	device_info.ring_size = ring_size;
+
 	rndis_filter_device_add(hdev, &device_info);
 	netif_tx_wake_all_queues(ndev);
 
@@ -803,6 +926,7 @@ static const struct ethtool_ops ethtool_ops = {
 	.get_drvinfo	= netvsc_get_drvinfo,
 	.get_link	= ethtool_op_get_link,
 	.get_channels   = netvsc_get_channels,
+	.set_channels   = netvsc_set_channels,
 };
 
 static const struct net_device_ops device_ops = {
@@ -876,6 +1000,8 @@ static int netvsc_probe(struct hv_device *dev,
 	int ret;
 	u32 max_needed_headroom;
 
+	memset(&device_info, 0, sizeof(device_info));
+
 	net = alloc_etherdev_mq(sizeof(struct net_device_context),
 				num_online_cpus());
 	if (!net)
diff --git a/drivers/net/hyperv/rndis_filter.c b/drivers/net/hyperv/rndis_filter.c
index 0d92efefd796..85091333c4c2 100644
--- a/drivers/net/hyperv/rndis_filter.c
+++ b/drivers/net/hyperv/rndis_filter.c
@@ -1011,7 +1011,7 @@ int rndis_filter_device_add(struct hv_device *dev,
 	unsigned long t;
 	struct ndis_recv_scale_cap rsscap;
 	u32 rsscap_size = sizeof(struct ndis_recv_scale_cap);
-	u32 mtu, size;
+	u32 mtu, size, max_chn;
 
 	rndis_device = get_rndis_device();
 	if (!rndis_device)
@@ -1096,12 +1096,19 @@ int rndis_filter_device_add(struct hv_device *dev,
 	ret = rndis_filter_query_device(rndis_device,
 					OID_GEN_RECEIVE_SCALE_CAPABILITIES,
 					&rsscap, &rsscap_size);
-	if (ret || rsscap.num_recv_que < 2)
+	max_chn = rsscap.num_recv_que;
+	if (ret || max_chn < 2)
 		goto out;
 
-	net_device->max_chn = rsscap.num_recv_que;
-	net_device->num_chn = (num_online_cpus() < rsscap.num_recv_que) ?
-			       num_online_cpus() : rsscap.num_recv_que;
+	net_device->max_chn = max_chn;
+
+	if (device_info->num_chn && device_info->num_chn < max_chn) {
+		net_device->num_chn = device_info->num_chn;
+	} else {
+		net_device->num_chn = (num_online_cpus() < max_chn) ?
+				       num_online_cpus() : max_chn;
+	}
+
 	if (net_device->num_chn == 1)
 		goto out;
 
@@ -1109,6 +1116,7 @@ int rndis_filter_device_add(struct hv_device *dev,
 					 NETVSC_PACKET_SIZE);
 	if (!net_device->sub_cb_buf) {
 		net_device->num_chn = 1;
+		/* should we set max_chn here too? */
 		dev_info(&dev->device, "No memory for subchannels.\n");
 		goto out;
 	}
-- 
2.4.0

Expo Deliverables

Here are the following deliverables for the 22nd Annual Senior Design Expo:

Senior Design Expo Deliverables
Document Description Link
Poster Display booth project design poster. File:2014 Ethtool Support Microsoft Hyper-V Expo Poster.pdf
Technical Presentation Slides Powerpoint slides used for the expo technical presentation. File:2014 Ethtool Support Microsoft Hyper-V Expo Presentation.pdf

Team Biographies

2014 Ethtool Schwartzmeyer.jpg

Andrew Schwartzmeyer

Computer Science and Mathematics Student

Hometown: Seattle, WA

Bio: Andrew is a senior undergraduate student in Computer Science and Mathematics at the University of Idaho. He is the current chair of the University of Idaho Association of Computing Machinery, and an avid open source developer. He blogs at schwartzmeyer.com, and is interested in Linux systems programming, evolutionary computation, security and cryptography, server administration automation (with Puppet), and functional programming.

Email: schw2620@vandals.uidaho.edu

2014 Ethtool Hall.jpg

Justin Hall

Computer Science Student

Hometown: Lewiston, ID

Bio: Justin is a senior in Computer Science at the University of Idaho, currently pursuing his Bachelors of Science. He spends summers as a freelance front-end Javascript developer, particularly the AngularJS framework. His interests focus on front-end and full-stack web development.

Email: hall5714@vandals.uidaho.edu

2014 Ethtool Drew.jpg

Keith Drew

Computer Science Student

Hometown: Olympia, Washington

Bio: Keith is a senior undergraduate student in Computer Science at the University of Idaho. He is also a NSF Scholarship for Service (SFS) CyberCorps Student. He is interested cyber security, specifically secure programming.

Email: keithd@vandals.uidaho.edu

2015 Ethtool Doman.png

Eric Doman

Computer Science Student

Hometown: St. Maries, Idaho

Bio: Eric is a senior undergraduate student in Computer Science at the University of Idaho. He is a member of the University of Idaho Association of Computing Machinery. He is interested in computer graphics and visualization. In his free time, Eric enjoys running and playing soccer.

Email: doma8101@vandals.uidaho.edu

2014 Ethtool Waltrip.png

Chris Waltrip (Fall 2014)

Computer Science Student

Hometown: Coeur d'Alene, ID

Bio: Chris is a senior undergraduate in Computer Science at the University of Idaho. He is also a NSF Scholarship for Service (SFS) CyberCorps Student and system administrator for the Center for Secure and Dependable Systems (CSDS). His interests lie in information assurance and network security.

Email: walt2178@vandals.uidaho.edu

Appendices

References

Other Links