COMP 541 Lecture Notes - Lecture 4: Weighted Arithmetic Mean, Raw Image Format, Mapreduce
Document Summary
Exercise 2. 2. 1 : suppose we execute the word-count mapreduce program described in this section on a large repository such as a copy of the web. We shall use 100 map tasks and some number of reduce tasks. (a) suppose we do not use a combiner at the map tasks. There will be significant skew since some keys will have large number of occurances [lengths of the value lists] while some have less occurances, so different reducers take different amount of time. We can take an example from real world dictionaries where the word distribibution follows power law. What if we instead combine the reducers into 10,000. The skew will be present but not as worse as in first case. After combination of reducers to some reduce tasks cause an averaging over execution times of several reducer tasks. Since words will be combined in the mapping phase only.
