COMP 541 Lecture Notes - Lecture 4: Weighted Arithmetic Mean, Raw Image Format, Mapreduce

1482 views3 pages
26 Dec 2017
School
Course
Professor

Document Summary

Exercise 2. 2. 1 : suppose we execute the word-count mapreduce program described in this section on a large repository such as a copy of the web. We shall use 100 map tasks and some number of reduce tasks. (a) suppose we do not use a combiner at the map tasks. There will be significant skew since some keys will have large number of occurances [lengths of the value lists] while some have less occurances, so different reducers take different amount of time. We can take an example from real world dictionaries where the word distribibution follows power law. What if we instead combine the reducers into 10,000. The skew will be present but not as worse as in first case. After combination of reducers to some reduce tasks cause an averaging over execution times of several reducer tasks. Since words will be combined in the mapping phase only.

Get access

Grade+20% OFF
$8 USD/m$10 USD/m
Billed $96 USD annually
Grade+
Homework Help
Study Guides
Textbook Solutions
Class Notes
Textbook Notes
Booster Class
40 Verified Answers
Class+
$8 USD/m
Billed $96 USD annually
Class+
Homework Help
Study Guides
Textbook Solutions
Class Notes
Textbook Notes
Booster Class
30 Verified Answers

Related Documents