Large Numbers of Devices
| We'll start by configuring a UML instance with a large number of devices. The reasons for wanting to do this vary. For many people, there is value in looking at /proc/meminfo and seeing an absurdly large amount of memory, or running df and seeing more disk space than you could fit in a room full of disks. More seriously, it allows you to explore the scalability limits of the Linux kernel and the applications running on it. This is useful when you are maintaining some software that may run into these limits, and your users may have hardware that may do so, but you don't. You can emulate the large configuration to see how your software reacts to it. You may also be considering acquiring a very large machine but want to know whether it is fully usable by Linux and the applications you envision running on it. UML will let you explore the software limitations. Obviously, any hardware limitations, such as the number of bus slots and controllers and the like, can't be explored in this way. Network InterfacesLet's start by configuring a pair of UML instances with a large number of network interfaces. We will boot the two instances, debian1 and debian2, and hot-plug the interfaces into them. So, with the UML instances booted, you do this as follows: host% for i in `seq 0 127`; do uml_mconsole debian1 \ config eth$i=mcast,,224.0.0.$i; done host% for i in `seq 0 127`; do uml_mconsole debian2 \ config eth$i=mcast,,224.0.0.$i; done These two lines of shell configure 128 network interfaces in each UML instance. You'll see a string of OK messages from each of these, plus a lot of console output in the UML instances if kernel output is logged there. Running dmesg in one of the instances will show you something like this: Netdevice 124 : mcast backend multicast address: \ 224.0.0.124:1102, TTL:1 Configured mcast device: 224.0.0.125:1102-1 Netdevice 125 : mcast backend multicast address: \ 224.0.0.125:1102, TTL:1 Configured mcast device: 224.0.0.126:1102-1 Netdevice 126 : mcast backend multicast address: \ 224.0.0.126:1102, TTL:1 Configured mcast device: 224.0.0.127:1102-1 Netdevice 127 : mcast backend multicast address: \ 224.0.0.127:1102, TTL:1 Running ifconfig inside the UML instances will confirm that interfaces eth0 through etH127 now exist. If you're brave, run ifconfig -a. Otherwise, just do some spot-checking: UML# ifconfig eth120 eth120 Link encap:Ethernet HWaddr 00:00:00:00:00:00 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 \ frame:0 TX packets:0 errors:0 dropped:0 overruns:0 \ carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) Interrupt:5 This indicates that we indeed have the network interfaces we asked for. I configured them to attach to multicast networks on the host, so they will be used purely to network between the two instances. They can't talk directly to the outside network unless you configure one of the instances with an interface attached to a TUN/TAP device and use it as a gate way. Each of an instance's interfaces is attached to a different host multicast address, which means they are on different networks. So, taken in pairs, the corresponding interfaces on the two instances are on the same network and can communicate with each other. For example, the two eth0 interfaces are both attached to the host multicast IP address 224.0.0.0 and thus will see each other's packets. The two etH1 interfaces are on 224.0.0.1 and can see each other's packets, but they won't see any packets from the eth0 interfaces. Next, we configure the interfaces inside the UML instances. I'm going to put each one on a different network in order to correspond to the connectivity imposed by the multicast configuration on the host. The eth0 interfaces will be on the 10.0.0.0/24 network, the etH1 interfaces will be on the 10.0.1.0/24 network, and so forth: UML1# for i in `seq 0 127`; do ifconfig eth$i 10.0.$i.1/24 up; done UML2# for i in `seq 0 127`; do ifconfig eth$i 10.0.$i.2/24 up; done Now the interfaces in the first UML instance are running and have the .1 addresses in their networks, and the interfaces in the second instance have the .2 addresses. Again, some spot-checking will confirm this: UML1# ifconfig eth75 eth75 Link encap:Ethernet HWaddr FE:FD:0A:00:4B:01 inet addr:10.0.75.1 Bcast:10.255.255.255 \ Mask:255.0.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 \ Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 \ frame:0 TX packets:0 errors:0 dropped:0 overruns:0 \ carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) Interrupt:5 UML2# ifconfig eth100 eth100 Link encap:Ethernet HWaddr FE:FD:0A:00:64:02 inet addr:10.0.100.2 Bcast:10.255.255.255 \ Mask:255.0.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 \ Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 \ frame:0 TX packets:0 errors:0 dropped:0 overruns:0 \ carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) Interrupt:5 Let's see if the interfaces work: UML1# ping 10.0.50.2 PING 10.0.50.2 (10.0.50.2): 56 data bytes 64 bytes from 10.0.50.2: icmp_seq=0 ttl=64 time=56.3 ms 64 bytes from 10.0.50.2: icmp_seq=1 ttl=64 time=15.7 ms 64 bytes from 10.0.50.2: icmp_seq=2 ttl=64 time=16.6 ms 64 bytes from 10.0.50.2: icmp_seq=3 ttl=64 time=14.9 ms 64 bytes from 10.0.50.2: icmp_seq=4 ttl=64 time=16.4 ms --- 10.0.50.2 ping statistics --- 5 packets transmitted, 5 packets received, 0% packet loss round-trip min/avg/max = 14.9/23.9/56.3 ms You can try some of the others by hand or check all of them with a bit of shell such as this: UML1# for i in `seq 0 127`; do ping -c 1 10.0.$i.2 ; done This exercise is fun and interesting, but what's the practical use? We have demonstrated that there appears to be no limit, aside from memory, on how many network interfaces Linux will support. To tell for sure, we would need to look at the kernel source. But if you are seriously asking this sort of question, you probably have some hardware limit in mind, and setting up some virtual machines is a quick way to tell whether the operating system or the networking tools have a lower limit. By poking around a bit more, we can see that other parts of the system are being exercised. Taking a look at the routing table will show you one route for every device we configured. An excerpt looks like this: UML1# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric \ Ref Use Iface 10.0.20.0 0.0.0.0 255.255.255.0 U 0 \ 0 0 eth20 10.0.21.0 0.0.0.0 255.255.255.0 U 0 \ 0 0 eth21 10.0.22.0 0.0.0.0 255.255.255.0 U 0 \ 0 0 eth22 10.0.23.0 0.0.0.0 255.255.255.0 U 0 \ 0 0 eth23 This would be interesting if you wanted a large number of networks, rather than simply a large number of interfaces. Similarly, we are exercising the arp cache more than usual. Here is an excerpt: UML# arp -an ? (10.0.126.2) at FE:FD:0A:00:7E:02 [ether] on eth126 ? (10.0.64.2) at FE:FD:0A:00:40:02 [ether] on eth64 ? (10.0.110.2) at FE:FD:0A:00:6E:02 [ether] on eth110 ? (10.0.46.2) at FE:FD:0A:00:2E:02 [ether] on eth46 ? (10.0.111.2) at FE:FD:0A:00:6F:02 [ether] on eth111 This all demonstrates that, if there are any hard limits in the Linux networking subsystem, they are reasonably high. A related but different question is whether there are any problems with performance scaling to this many interfaces and networks. If you are concerned about this, you probably have a particular application or workload in mind and would do well to run it inside a UML instance, varying the number of interfaces, networks, routes, or whatever its performance depends on. For demonstration purposes, since I lack such a workload, I will use standard system tools to see how well performance scales as the number of interfaces increases. Let's look at ping times as the number of interfaces increases. I'll shut down all of the Ethernet devices and bring up an increasing number on each test. The first two rounds look like this: UML# export n=0 ; for i in `seq 0 $n`; \ do ifconfig eth$i 10.0.$i.1/24 up; done ; \ for i in `seq 0 $n`; do ping -c 2 10.0.$i.2 ; done ; \ for i in `seq 0 $n`; do ifconfig eth$i down ; done PING 10.0.0.2 (10.0.0.2): 56 data bytes 64 bytes from 10.0.0.2: icmp_seq=0 ttl=64 time=36.0 ms 64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=4.9 ms --- 10.0.0.2 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 4.9/20.4/36.0 ms UML# export n=1 ; for i in `seq 0 $n`; \ do ifconfig eth$i 10.0.$i.1/24 up; done ; \ for i in `seq 0 $n`; do ping -c 2 10.0.$i.2 ; \ done ; for i in `seq 0 $n`; do ifconfig eth$i down ; done PING 10.0.0.2 (10.0.0.2): 56 data bytes 64 bytes from 10.0.0.2: icmp_seq=0 ttl=64 time=34.0 ms 64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=4.9 ms --- 10.0.0.2 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 4.9/19.4/34.0 ms PING 10.0.1.2 (10.0.1.2): 56 data bytes 64 bytes from 10.0.1.2: icmp_seq=0 ttl=64 time=35.4 ms 64 bytes from 10.0.1.2: icmp_seq=1 ttl=64 time=5.0 ms --- 10.0.1.2 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 5.0/20.2/35.4 ms The two-interface ping times are essentially the same as the one-interface times. We are looking at how the times change, rather than their actual values compared to ping times on the host. A virtual machine will necessarily have different performance characteristics than a physical one, but they should scale similarly. We see the first ping taking much longer than the second because of the arp request and response that have to occur before any ping requests can be sent out. The sending system needs to determine the Ethernet MAC address corresponding to the IP address you are pinging. This requires an arp request to be broadcast and a reply to come back from the target host before the actual ping request can be sent. The second ping time measures the actual time of a ping round trip. I won't bore you with the full output of repeating this, doubling the number of interfaces at each step. However, this is typical of the times I got with 128 interfaces: 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 6.7/22.7/38.8 ms PING 10.0.123.2 (10.0.123.2): 56 data bytes 64 bytes from 10.0.123.2: icmp_seq=0 ttl=64 time=39.1 ms 64 bytes from 10.0.123.2: icmp_seq=1 ttl=64 time=8.9 ms --- 10.0.123.2 ping statistics --- With 128 interfaces, both ping times are around 4 ms greater than with one. This suggests that the slowdown is in the IP routing code since this is exercised once for each packet. The arp requests don't go through the IP stack, so they wouldn't be affected by any slowdowns in the routing code. The 4-ms slowdown is comparable to the fastest ping time, which was around 5 ms, suggesting that the routing overhead with 128 networks and 128 routes is comparable to the ping round trip time. In real life, you're unlikely to be interested in how fast pings go when you have a lot of interfaces, routes, arp table entries, and so on. You're more likely to have a workload that needs to operate in an environment with these sorts of scalability requirements. In this case, instead of running pings with varying numbers of interfaces, you'd run your workload, changing the number of interfaces as needed, and make sure it behaves acceptably within the range you plan for your hardware. MemoryMemory is another physical asset that a system may have a lot of. Even though it's far cheaper than it used to be, outfitting a machine with many gigabytes is still fairly pricy. You may still want to emulate a large-memory environment before splashing out on the actual physical article. Doing so may help you decide whether your workload will benefit from having lots of memory, and if so, how much memory you need. You can determine your memory sweet spot so you spend enough on memory but not too much. You may have guessed by now that we are going to look at large-memory UML instances, and you'd be right. To start with, here is /proc/meminfo from a 64GB UML instance: UML# more /proc/meminfo MemTotal: 65074432 kB MemFree: 65048744 kB Buffers: 824 kB Cached: 9272 kB SwapCached: 0 kB Active: 5252 kB Inactive: 6016 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 65074432 kB LowFree: 65048744 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 112 kB Writeback: 0 kB Mapped: 2772 kB Slab: 4724 kB CommitLimit: 32537216 kB Committed_AS: 4064 kB PageTables: 224 kB VmallocTotal: 137370258416 kB VmallocUsed: 0 kB VmallocChunk: 137370258416 kB This output is from an x86_64 UML on a 1GB host. Since x86_64 is a 64-bit architecture, there is plenty of address space for UML to map many gigabytes of physical memory. In contrast, x86, as a 32-bit architecture, doesn't have sufficient address space to cleanly handle large amounts of memory. On x86, UML must use the kernel's Highmem support in order to handle greater than about 3GB of physical memory. This works, but, as I discussed in Chapter 9, there's a large performance penalty to pay because of the requirement to map the high memory into low memory where the kernel can directly access it. On an x86 UML instance, the meminfo output would have a large amount of Highmem in the HighTotal and HighFree fields. On 64-bit hosts, this is unnecessary, and all the memory appears as LowTotal and LowFree. The other unusual feature here is the even larger amount of vmalloc space, 137 terabytes. This is simply the address space that the UML instance doesn't have any other use for. There has to be more merit to large-memory UML instances than impressive numbers in /proc/meminfo. That's enough for me, but other people seem to be more demanding. A more legitimate excuse for this sort of exercise is to see how the performance of a workload or application will change when given a large amount of memory. In order to do this, we need to be able to judge the performance of a workload in a given amount of memory. On a physical machine, this would be a matter of running it and watching the clock on the nearest wall. Having larger amounts of memory improves performance by allowing more data to be stored in memory, rather than on disk. With insufficient memory, the system has to swap data to disk when it's unused and swap it back in when it is referenced again. Some intelligent applications, such as databases, do their own caching based on the amount of memory in the system. In this case, the trade-off is usually still against storing data in memory. For example, a database will read more index data from disk when it has enough memory, speeding lookups. In the example above, the 64GB UML instance is running on a 1GB host. It's obviously not manufacturing 63GB of memory, so that extra memory is ultimately backed by disk. You can run applications that consume large amounts of memory, and the UML instance will not have to use its own swap. However, since this will exceed the amount of memory on the host, it will start swapping. This means you can't watch the clock in order to decide how your workload will perform with a lot of memory available. Instead, you need to find a proxy for performance. A proxy is a measurement that can stand in for the thing you are really interested in when that thing can't be measured directly. I've been talking about disk I/O, either by the system swapping or by the application reading in data on its own. So, watching the UML instance's disk I/O is a good way to decide whether the workload's performance will improve. The greater the decrease in disk traffic, the greater the performance improvement you can expect. As with increasing amounts of any resource, there will be a point of diminishing returns, where adding an increment of memory results in a smaller performance increase than the previous increment did. Graphing performance against memory will typically show a relatively narrow region where the performance levels off. It may still increase, but suddenly at a slower rate than before. This performance "knee" is usually what you aim at when you design a system. Sometimes the knee is too expensive or is unattainable, and you add as much memory as you can, accepting a performance point below the knee. In other cases, you need as much performance as you can get, and you accept the diminishing performance returns with much of the added memory. As before, I'm going to use a little fake workload in order to demonstrate the techniques involved. I will create a database-like workload with a million small files. The file metadatathe file names, sizes, modification dates, and so onwill stand in for the database indexes, and their contents will stand in for the actual data. I need such a large number of files so that their metadata will occupy a respectable amount of memory. This will allow us to measure how changing the amount of system memory impacts performance when searching these files. The following procedure creates the million files in three stages, increasing the number by a factor of 100 at each step:
UML# mkdir test UML# cd test UML# dd if=/etc/passwd count=1024 bs=1 > 0 1024+0 records in 1024+0 records out UML# for n in `seq 99` ; do cp 0 $n; done UML# ls 1 14 19 23 28 32 37 41 46 50 55 6 \ 64 69 73 78 82 87 91 96 10 15 2 24 29 33 38 42 47 51 56 60 \ 65 7 74 79 83 88 92 97 11 16 20 25 3 34 39 43 48 52 57 61 \ 66 70 75 8 84 89 93 98 12 17 21 26 30 35 4 44 49 53 58 62 \ 67 71 76 80 85 9 94 99 13 18 22 27 31 36 40 45 5 54 59 63 \ 68 72 77 81 86 90 95 0 UML# mkdir a UML# mv * a mv: cannot move `a' to a subdirectory of itself, `a/a' UML# mv a 0 UML# for n in `seq 99` ; do cp -a 0 $n; done UML# mkdir a UML# mv * a mv: cannot move `a' to a subdirectory of itself, `a/a' UML# mv a 0 UML# for n in `seq 99` ; do cp -a 0 $n; done Now let's reboot in order to get some clean memory consumption data. On reboot, log in, and look at /proc/diskstats in order to see how much data was read from disk during the boot: UML# cat /proc/diskstats 98 0 ubda 375 221 18798 2860 55 111 1328 150 0 2740 3010 The sixth field (18798, in this case) is the number of sectors read from the disk so far. With 512-byte sectors, this means that the boot read around 9.6MB (9624576 bytes, to be exact). Now, to see how much memory we need in order to search the metadata of the directory hierarchy, let's run a find over it: UML# cd test UML# find. > /dev/null Let's look at diskstats again, using awk to pick out the correct field so as to avoid taxing our brains by having to count up to six: UML# awk '{ print $6 }' /proc/diskstats 214294 UML# echo $[ (214214 - 18798) * 512 ] 100052992This pulled in about 100MB of disk space. Any amount of memory much more than that will be plenty to hold all of the metadata we will need. To check this, we can run the find again and see that there isn't much disk input: UML# awk '{ print $6 }' /proc/diskstats 215574 UML# find. > /dev/null UML# awk '{ print $6 }' /proc/diskstats 215670So, there wasn't much disk I/O, as expected. To see how much total memory would be required to run this little workload, let's look at /proc/meminfo: UML# grep Mem /proc/meminfo MemTotal: 1014032 kB MemFree: 870404 kB A total of 143MB of memory has been consumed so far. Anything over that should be able to hold the full set of metadata. We can check this by rebooting with 160MB of physical memory: UML# cd test UML# awk '{ print $6 }' /proc/diskstats 18886 UML# find . > /dev/null UML# awk '{ print $6 }' /proc/diskstats 215390 UML# find . > /dev/null UML# awk '{ print $6 }' /proc/diskstats 215478 UML# grep Mem /proc/meminfo MemTotal: 156276 kB MemFree: 15684 kBThis turns out to be correct. We had essentially no disk reads on the second search and pretty close to no free memory afterward. We can check this by booting with a lot less memory and seeing if there is a lot more disk activity on the second find. With an 80MB UML instance, there was about 90MB of disk activity between the two searches. This indicates that 80MB was not enough memory for optimal performance in this case, and a lot of data that was cached during the first search had to be discarded and read in again during the second. On a physical machine, this would result in a significant performance loss. On a virtual machine, it wouldn't necessarily, depending on how well the host is caching data. Even if the UML instance is swapping, the performance loss may not be nearly as great as on a physical machine. If the host is caching the data that the UML instance is swapping, then swapping the data back in to the UML instance involves no disk activity, in contrast to the case with a physical machine. In this case, swapping would result in a performance loss for the UML instance, but a lot less than you would expect for a physical system. We measured the difference between an 80MB UML instance and a 160MB one, which are very far from the 64MB instance with which I started. These memory sizes are easily reached with physical systems today (it would be hard to buy a system with less than many times as much memory as this), and this difference could easily have been tested on a physical system. To get back into the range of memory sizes that aren't so easily reached with a physical machine, we need to start searching the data. My million files, plus the rest of the files that were already present, occupy about 6.5GB. With a 1GB UML instance, there are about 5.5GB of disk I/O on the first search and about the same on the second, indicating that this is not nearly enough memory and that there is less actual data being read from the disk than df would have us believe: UML# awk '{ print $6 }' /proc/diskstats 18934 UML# find . -xdev -type f | xargs cat > /dev/null UML# awk '{ print $6 }' /proc/diskstats 11033694 UML# find . -xdev -type f | xargs cat > /dev/null UML# awk '{ print $6 }' /proc/diskstats 22050006 UML# echo $[ (11033694 - 18934) * 512 ] 5639557120 UML# echo $[ (22050006 - 11033694) * 512 ] 5640351744With a 4GB UML instance, we might expect the situation to improve, but with still a noticeable amount of disk activity on the second search. UML# awk '{ print $6 }' /proc/diskstats 89944 UML# find / -xdev -type f | xargs cat > /dev/null UML# awk '{print $6}' /proc/diskstats 13187496 UML# echo $[ 13187496 * 512 ] 6751997952 UML# awk '{print $6}' /proc/diskstats 26229664 UML# echo $[ (26229664 - 13187496) * 512 ] 6677590016Actually, there is no improvementthere was just as much input during the second search as during the first. In retrospect, this shouldn't be surprising. While a lot of the data could have been cached, it wasn't because the kernel had no way to know that it was going to be used again. So, the data was thrown out in order to make room for data that was read in later. In situations like this, the performance knee is very sharpyou may see no improvement with increasing memory until the workload's entire data set can be held in memory. At that point, there will likely be a very large performance improvement. So, rather than the continuous performance curve you might expect, you would get something more like a sudden jump at the magic amount of memory that holds all of the data the workload will need. We can check this by booting a UML instance with more than about 6.5GB of memory. Here are the results with a 7GB instance: UML# awk '{print $6}' /proc/diskstats 19928 UML# find / -xdev -type f | xargs cat > /dev/null UML# awk '{print $6}' /proc/diskstats 13055768 UML# echo $[ (13055768 - 19928) * 512 ] 6674350080 UML# find / -xdev -type f | xargs cat > /dev/null UML# awk '{print $6}' /proc/diskstats 14125882 UML# echo $[ (14125882 - 13055768) * 512 ] 547898368We had about a half gigabyte of data read in from disk on the second run, which I don't really understand. However, this is far less than we had with the smaller memory instances. On a physical system, this would have translated into much better performance. The UML instance didn't run any faster with more memory because real time is going to depend on real resources. The real resource in this case is physical memory on the host, which was the same for all of these tests. In fact, the larger memory instances performed noticeably worse than the smaller ones. The smallest instance could just about be held in the host's memory, so its disk I/O was just reading data on behalf of the UML instance. The larger instances couldn't be held in the host's memory, so there was that I/O, plus the host had to swap a large amount of the instance itself in and out. This emphasizes the fact that, in measuring performance as you adjust the virtual hardware, you should not look at the clock on the wall. You should find some quantity within the UML instance that will correlate with performance of a physical system with that hardware running the same workload. Normally, this is disk I/O because that's generally the source for all the data that's going to fill your memory. However, if the data is coming from the network, and increasing memory would be expected to reduce network use, then you would look at packet counts rather than disk I/O. If you were doing this for real in order to determine how much memory your workload needs for good performance, you wouldn't have created a million small files and run find over them. Instead, you'd copy your actual workload into a UML instance and boot it with varying amounts of memory. A good way to get an approximate number for the memory it needs is to boot with a truly large amount of memory, run the workload, and see how much data was read from disk. A UML instance with that amount of memory, plus whatever it needs during boot, will very likely not need to swap out any data or read anything twice. However, this approximation may overstate the amount of memory you need for decent performancea good amount of it may be holding data that is not important for performance. So, it would also be a good idea, after checking this first amount of memory to see that it gives you good performance, to decrease the memory size until you see an increase in disk reads. At this point, the UML instance can't hold all of the data that is needed for good performance. This, plus a bit more, is the amount of memory you should aim at with your physical system. There may be reasons it can't be reached, such as it being too expensive or the system not being able to hold that much. In this case, you need to accept lower than optimal performance, or take some more radical steps such as reworking the application to require less memory or spreading it across several machines, as with a cluster. You can use UML to test this, as well. |
