13.5 Experimental analysis and conclusion

< Free Open Study >

This section includes the intermediate analysis of the experimental results. These results compare each workload in individual scenarios. Based on these results the final conclusion regarding the tradeoff between the operating systems concludes this chapter.

13.5.1 File transfer workload

For the file transfer workload, when the file size is constant and the number of files is increased, the response time and CPU utilization increase linearly, whereas memory utilization remains almost constant for XP, NT, and ME. But in the case of LINUX, the response time increases, but the CPU utilization decreases and the memory utilization remains constant.

Tables 13.5 through 13.7 show the ranges of CPU use and memory utilization, depending on the number of files for each OS.

Observations for Table 13.5: Windows ME, NT, and XP all increased their CPU use ranges, while LINUX decreased its range. LINUX performance fell within the range of 75.6-99.9 for 100 files. For file numbers of 500 and 1,000 the CPU use was between 13 and 16.8 with a varying trend.

Table 13.5: CPU Use and Memory Utilization for File Size = 500
Operating System	CPU Range	Memory Utilization
ME	56.5-88.8	10
NT	87.37-98.08	10
LINUX	13.0-99.9	0.3
XP	13.92-26.7	18

Observations for Table 13.6: Windows ME, NT, and XP all increased their CPU utilization ranges, while LINUX decreased its utilization range; overall measurements varied greatly in the case of 100 files. CPU utilization measurements varied from 98.3 for the first experiment, 24.4 for the second experiment, and 99.9 for the third experiment. However, for 500 and 1,000 files it ranged from 13.7 to 15.9, showing less variability. Some of this variability could be smoothed out by performing more experiments at each level and averaging their results. The time allotted to our experiment did not allow us to do this, however.

Table 13.6: CPU Use and Memory Utilization For File Size = 750
Operating System	CPU Range	Memory Utilization
ME	62.0-87.8	10
NT	6.35-98.49	10
LINUX	13.7-99.9	0.4
XP	16.16-32.77	17.99

Observations for Table 13.7: Windows ME, NT, and XP all increased in their CPU utilization ranges over the full spectrum of measurements, while LINUX decreased overall, staying at about 98 percent for the three experimental runs. However, for 500 and 1,000 files its CPU utilization ranged from 12.9 to 15.6.

Table 13.7: CPU Use and Memory Utilization for File Size = 1,000
Operating System	CPU Range	Memory Utilization
ME	60.0-87.8:	10
NT	96.11-97.68	10-11
LINUX:	12.9-98.2	0.5
XP	18.156-30.91	17.98

13.5.2 Process creation workload

For the process creation workload, we realize that the CPU utilization is approximately linear for XP and LINUX with an increasing slope, while NT and ME vary widely in their CPU utilization. Also, for XP, the response time decreases when the number of processes rate increases. For each number of processes the performance range is higher than the previous number, but it decreases as the processes rate increases.

Tables 13.8 through 13.10 show the ranges of CPU utilization and memory utilization depending on the process rates for each OS.

Observations for Table 13.8: The Windows NT CPU utilization is approximately 22 percent for ten processes but increases to between 77 percent and 85 percent for 100 processes and drops down to about 50 percent for 1,000 processes. Also, for 1,000 processes, the memory utilization increases even though the CPU utilization decreases.

Table 13.8: CPU Use and Memory Utilization for Process Rate = 0
Operating System	CPU Range	Memory Utilization
ME	61.67-91.2	10-12
NT:	3.38-85.625	11-26
LINUX	25-75	0.1
XP	47-86	17.23-18

Observations for Table 3.9: For the case of a process rate of 100, NT generally increases linearly with the exception of some values where it happens to use a little less CPU utilization. However, the memory utilization seems to increase linearly. For LINUX the process rate stays in the range of 37 to 41 for a number of processes equaling 10 and 100, but for 1,000 the utilization jumps to between 50 percent and 70 percent.

Table 13.9: CPU Use and Memory Utilization for Process Rate = 100
Operating System	CPU Range	Memory Utilization
ME	60.05-73.25:	10-14.56
NT	12.33-31.59	11-15
LINUX	37.1-73.3	0.1
XP	77.3-85.27	17.98-18.99

Table 13.10: CPU Use and Memory Utilization for Process Rate = 1,000
Operating System	CPU Range	Memory Utilization
ME	56.13-58.85	1.322-11
NT	3.42-4.62	11-20.98
LINUX	41.6-74.1	0.1
XP	18.156-30.91	17.97-17.99

Observations for Table 3.10: NT and XP remained almost constant in their CPU, with a fluctuation of about 3 percent to 4 percent. The only OS showing a variation in the percentage of CPU utilization was LINUX, ranging from 41.6 to 74.1. On the contrary, ME showed a decrease in CPU utilization with the increase in the number of processes. As the number of processes increased, the memory utilization also increased considerably (maximum of 8 percent in NT).

13.5.3 MATLAB workload

For the matrix operations for the MATLAB workload, with constant matrix size and varying number of matrices, the response time and CPU utilization in the case of Windows ME, NT, and XP increase, whereas the memory utilization remains almost constant. (See Tables 13.11 through 13.14.)

Observations for Table 13.11: Windows NT and XP performed similarly, with CPU utilization exponentially increasing when the number of matrices was increased from 10 to 1,000. The variation was a bit less in ME, ranging from 51 percent to 77 percent. Even for ten matrices it consumed a lot of computational power. For the LINUX OS, the experiment was conducted with different parameters. The matrix size started at 50 and went up to 1,000, while for the Windows-based OS it started at 10 and went up to 100. The amount of memory consumed was considerably less in the ME system (2.5 percent) as compared with the other Windows-based operating systems. On the other hand, the percentage of variation was fairly constant in all three of them.

Table 13.11: CPU Use and Memory Utilization for Matrix Size = 10 × 10
Operating System	CPU Range	Memory Utilization
ME	51.5-76.66	2-2.5
NT	2-74	16
LINUX	N/A	N/A
XP	4.5-56.67	17.97-17.99

Observations for Table 13.12: Percentage utilization of CPU for NT and XP increased exponentially when the number of matrices was increased from 10 to 1,000. The variation was a bit less in ME, ranging from 55 percent to 100 percent. Considerably, ME consumed more CPU even for a lesser number of matrices. For the LINUX OS, the CPU consumption didn't vary much but still consumed a lot of computational power for a lesser number of matrices, similar to ME. The amount of memory consumed was considerably less in the LINUX operating system (4.2 percent) as compared with the Windows-based operating systems. On the other hand, the percentage of variation in performance was fairly constant in all four of them.

Table 13.12: CPU Use and Memory Utilization for Matrix Size = 50×50
Operating System	CPU Range	Memory Utilization
ME	55-99.78	4-11.97
NT	14-99.62	16-18
LINUX	46-48.37	4.2
XP	7.66-99.17	28-30.97

Observations for Table 13.13: All three of the Windows-based machines saturated their CPU utilization when the number of matrices was about 1,000. In this scenario too, Windows ME consumed more CPU resources even for small numbers of matrices. For the LINUX operating system, the CPU consumption didn't vary much but still consumed a significant amount of computational power for small numbers of matrices, similar to ME. The amount of memory consumed was considerably less in the LINUX operating system (4.2 percent) as compared with the other Windows-based OSs.

Table 13.13: CPU Use and Memory Utilization for Matrix Size = 100 × 100
Operating System	CPU Range	Memory Utilization
ME	59-99.98	0.30-4.25
NT	39.50-100	16-24.98
LINUX	81.22-81.97	4.2
XP	7.3-100	28-41

Observations for Table 13.14: Only the LINUX team tested a matrix of this size. In this case the CPU utilization was almost pushed to the limit. The memory utilization was still less compared with the Windows-based machines.

Table 13.14: CPU Use and Memory Utilization for Matrix Size = 1,000 × 1,000
Operating System	CPU Range	Memory Utilization
LINUX	96.76-96.95	17.95-19.5

13.5.4 Final conclusion

The following tables were deduced by calculating the average statistical measured values for the actual tables collected in the study for the different operating systems. The measure used is a ratio of the CPU utilization divided by the product of the memory use and overall response time for each experiment.

Table 13.15 gives the results for the process experiments.

Table 13.15: Results for the Process Experiments
Operating System	Response Time (ms)	% CPU Utilization	% Memory Utilization	*CPU/(Memres)**
XP	35,290.95	82.34	19.43	12E-05
ME	150,110.33	68.98	8.96	5.13E-05
NT	148,956.52	27.03	13.91	1.3E-05
LINUX	7,400.5	49.83	0.1	6733.3E-05

Table 13.16 gives the results for the MATLAB experiments.

Table 13.16: Results for the MATLAB Experiments
Operating System	Response Time (ms)	% CPU Utilization	% Memory Utilization	CPU/(Mem ^[]res)*
XP	218.66	99.56	35.975	0.012657
ME	224.555	99.715	26.45	0.016789
NT	222.345	98.7	21.46	0.020685
LINUX	-	65.17	4.2	0.069941^[*]
^[*]In order to get a value for this table it was necessary to have a response time for LINUX. The response time for the other operating systems was averaged; this way the LINUX response time essentially doesn't come into play.

Table 13.17 gives the results for the file experiments.

Table 13.17: Results for the File Experiments
Operating System	Response Time (ms)	% CPU Utilization	% Memory Utilization	*CPU/(Memres)**
XP	164,194	67.74	17.95	2.29839E-05
ME	965.36	74.05	10	767.0714E-05
NT	129,030	96.06	10.29	7.23497E-05
LINUX	42,625.88	89.28	22.6	9.26771E-05

No operating system dominates in performance for all the workloads that were used in this study. Each of the operating systems outperforms other operating systems in its own way. To determine which system performs best for each workload, we used the formula in the final columns, which is equal to the CPU utilization divided by the product of the memory utilization and the response time.

LINUX performs well in forking new processes. This can be deduced from the table values for process experiments, where it utilizes the least memory, average CPU utilization, and minimum response time as compared with other operating systems, giving it the best value in the aggregate measure for performance. The second best system based on this measure is the XP operating system, which has a value worse than LINUX's by a factor of 561. Next comes NT, which has a value worse than XP's by a factor of 9.2. Finally, ME is the worst, with a value that is off by a factor of 3.9 when compared with NT.

For the experiments involving matrix operations in MATLAB, LINUX out performs the other operating systems, since it utilizes the least amount of memory and has an average CPU utilization with an aggregate performance measure of 0.069941, making it better than NT by a factor of 3.38. NT is better than ME by a factor of 1.23, and ME is better than XP by a factor of 1.33.

Windows ME manages files efficiently compared with the other operating systems, with an aggregate performance measure of 7.67E-03. This measure is then followed by LINUX, which is found to perform worse by a factor of 82.7. NT in turn performs worse than LINUX by a factor of 1.28, and XP is found to be worse than NT by a factor of 3.14.

13.5.5 Tabular results

Tables 13.18 through 13.20 show results for the various workloads.

Table 13.18: Intermediate Data for Workload 1
Files		Windows XP			Windows ME			Windows NT			LINUX
File Size (KB)	Number of Files	RT	CPU	MEM	RT	CPU	MEM	RT	CPU	MEM	RT	CPU	MEM
500	100	39,707	15.26	18	125.33	60.33	10	16,490.34	89.96	10	928.67	91.4	12
500	500	116,661	208	17.94	786.67	75.08	10	80,309	95.64	10	30,176.67	15.73	0.3
500	1,000	175,899	26.4	17.99	1,985.3	87.6	10	161,582.3	97.83	10	57,875	14.73	0.3
750	100	55,589.7	62.83	17.99	124.66	62.83	10	22,121.67	97.75	10	2,450.67	74.2	0.4
750	500	15,743.4	237	17.99	790	75.47	10	120,286	96.77	10	40,080	15.4	0.4
750	1,000	227,373.7	30.6	17.9	1,974.67	84.65	10	244,661.7	96.65	10	85,100	14.5	0.4
1,000	100	663,324	18.5	17.97	126.33	61.16	10	30,433.67	96.71	10.98	1,654.3	97.73	14.7
1,000	500	180,489.3	26.6	17.9	793	76.09	10	160,618	96.59	10.98	56,393	14.83	82.6
1,000	100	2,958.7	30.9	17.9	1,982.3	83.29	10	324,767	96.67	10.67	108,974.6	15	92.3

Table 13.19: Intermediate Data for Workload 2
Processes		Windows XP			Windows ME			Windows NT			LINUX
No. of Processes	Process Rate	RT	CPU	MEM	RT	CPU	MEM	RT	CPU	MEM	RT	CPU	MEM
10	0	607.3	47.91	17.28	271	63.78	10	307.3	22.3	11	212.6	43.4	0.1
10	100	584	63.165	18	1,095	71.08	10	1,217	15.51	11	206	35.93	0.1
10	1,000	630.67	61.45	17.23	1,000,063	58.9	10	10,235	3.18	11	122	66.67	0.1
100	0	23,936	106.44	18.55	2,072.3	87.3	11.53	3,445	79.94	12	2,315.3	41	0.1
100	100	4,212.67	83.45	17.99	10,502.6	65.99	10.67	122,551.34	28.78	11	2,308	38.1	0.1
100	1,000	4,613.34	86.76	17.99	10,0513	56.82	11	10,361.67	3.82	11.9	1,335.7	64.67	0.1
1,000	0	141,076.34	96.72	24	24,422	98.7	1.58	34,522.67	55	24.22	23,329	42.3	0.1
1,000	100	76,894	97.09	20.98	105,878	61.64	13.27	130,361	30.39	13.80	23,361.3	42.6	0.1
1,000	1,000	65,064.3	98.1	22.91	1,006,176	56.69	2.647	1,027,607.67	4.4	19.3	13,414.7	73.83	0.1

Table 13.20: Intermediate Data for Workload 3
Matrix		Windows XP			Windows ME			Windows NT			LINUX
Matrix Size	Matrix Rate	RT	CPU	MEM	RT	CPU	MEM	RT	CPU	MEM	RT	CPU	MEM
10×10	10	0.01	5.7	29	0.0	58.83	2	0.015	12	16	0.003	-	-
10×10	100	0.057	9.83	28	0.056	53.5	2	0.06	13.33	16	0.032	-	-
10×10	1,000	0.791	47.72	28	0.84	73.63	2.44	0.898	71	16	0.538	-	-
50×50	10	0.07	8.9	28	0.07	56.5	4	0.063	19.33	16	0.06	47.0	4.1
50×50	100	0.577	42.33	28	0.68	72.36	4.33	0.6403	77.5	16	0.68	47.89	4.1
50×50	1,000	48.55	99.12	30.97	50.37	99.46	11.9	49.93	99.46	17.99	50.75	48.376	4.2
100×100	10	0.324	34.07	28	0.33	62.5	2.33	0.32	38.9	16	0.61	81.34	4.2
100×100	100	2.92	79.92	28.87	3.46	89.30	4.08	3.14	19.08	16.8	5.2	81.76	4.2
100×100	1,000	388.77	100	40.98	398.74	99.97	41	394.76	97.95	24.93	417.7	81.97	4.2

< Free Open Study >