Hadoop留学生讲解、辅导Java，c/c++编程语言、讲解Python设计、辅导System 辅导Database|辅导Web开发
- 首页 >> OS编程 Exercise 1 Suspected Pairs (10 points)
Using the information from the first lecture (or Section 1.2.3 in the textbook), what would be the number of suspected pairs if the following changes were made to the data (all changes should be applied at once).
1.The number of days of observation was raised to 5000.
2.The number of people observed was raised to 5 billion (and there were therefore 500,000 hotels).
3.We only reported a pair as suspect if they were at the same hotel at the same time on four different days.
Exercise 2 Hadoop (10+10 points)
For this exercise, you have to set up and configure your system to use Hadoop. Follow the instructions in Stanford document at http://snap.stanford.edu/ class/cs246-2017/homeworks/hw0/tutorialv3.pdf and set up the virtual machine as described in Section 1. Run the example program of Section 2 and carry out the different steps given in that section.
The number of days of observation was raised to 5000.
Run your job on the file http://www.gutenberg.org/files/100/100-0. txt in standalone mode and pseudo-distributed mode and record the out- put.
Exercise 3 Friend Recommendation System (Stanford) (35 points)
Write a MapReduce program in Hadoop that implements a simple People You Might Know social network friendship recommendation algorithm. The key idea is that if two people have a lot of mutual friends, then the system should rec- ommend that they connect with each other. You have to run the program on the system setup in Exercise 2 in order to receive points for this exercise.
Input: Download the input file from the link: http://snap.stanford.edu/ class/cs246-data/hw1q1.zip. The input file contains the adjacency list and has multiple lines in the following format:
Here, is a unique integer ID corresponding to a unique user and is a comma separated list of unique IDs corresponding to the friends of the user with the unique ID . Note that the friendships are mutual (i.e., edges are undirected): if A is friend with B then B is also friend with A. Algorithm: Let us use a simple algorithm such that, for each user U, the algorithm recommends N = 10 users who are not already friends with U, but have the most number of mutual friends in common with U.
Output: The output should contain one line per user in the following for- mat:
where is a unique ID corresponding to a user and is a comma separated list of unique IDs corresponding to the algorithms recom- mendation of people that might know, ordered in decreasing number of mutual friends. Even if a user has less than 10 second-degree friends, output all of them in decreasing order of the num- ber of mutual friends. If there are recommended users with the same number of mutual friends, then output those user IDs in numerically ascending order. Also, please provide a description of how you are going to use MapReduce jobs to solve this problem. Do not write more than 3 to 4 sentences for this: we only want a very high-level description of your strategy to tackle this problem. Note: It is possible to solve this question with a single MapReduce job. But if your solution requires multiple map reduce jobs, then that is fine too.
For your submission?
Include your source code
Include in your writeup a short paragraph describing your algorithm to tackle this problem.
Include in your writeup the recommendations for the users with following user IDs: 924, 8941, 8942, 9019, 9020, 9021, 9022, 9990, 9992, 9993.
Using the information from the first lecture (or Section 1.2.3 in the textbook), what would be the number of suspected pairs if the following changes were made to the data (all changes should be applied at once).
1.The number of days of observation was raised to 5000.
2.The number of people observed was raised to 5 billion (and there were therefore 500,000 hotels).
3.We only reported a pair as suspect if they were at the same hotel at the same time on four different days.
Exercise 2 Hadoop (10+10 points)
For this exercise, you have to set up and configure your system to use Hadoop. Follow the instructions in Stanford document at http://snap.stanford.edu/ class/cs246-2017/homeworks/hw0/tutorialv3.pdf and set up the virtual machine as described in Section 1. Run the example program of Section 2 and carry out the different steps given in that section.
The number of days of observation was raised to 5000.
Run your job on the file http://www.gutenberg.org/files/100/100-0. txt in standalone mode and pseudo-distributed mode and record the out- put.
Exercise 3 Friend Recommendation System (Stanford) (35 points)
Write a MapReduce program in Hadoop that implements a simple People You Might Know social network friendship recommendation algorithm. The key idea is that if two people have a lot of mutual friends, then the system should rec- ommend that they connect with each other. You have to run the program on the system setup in Exercise 2 in order to receive points for this exercise.
Input: Download the input file from the link: http://snap.stanford.edu/ class/cs246-data/hw1q1.zip. The input file contains the adjacency list and has multiple lines in the following format:
Here,
Output: The output should contain one line per user in the following for- mat:
where
For your submission?
Include your source code
Include in your writeup a short paragraph describing your algorithm to tackle this problem.
Include in your writeup the recommendations for the users with following user IDs: 924, 8941, 8942, 9019, 9020, 9021, 9022, 9990, 9992, 9993.