Tagged: mogilefs Toggle Comment Threads | Keyboard Shortcuts

  • wangxinxi 00:46 on February 26, 2011 Permalink | Reply
    Tags: hadoop, mogilefs,   

    The response time of MogileFS is much better than HDFS. And it can also hold more small files than HDFS. Using MogileFS as the streaming server is a good choice.

    Advertisements
     
  • wangxinxi 17:36 on February 21, 2011 Permalink | Reply
    Tags: development, hdfs, mogilefs,   

    MogileFS 

    Required by my research project, I need to setup a distributed file system which can store hundreds terabytes of music mp3 files while still remains highly robust and efficient.

    Since the number of mp3 files is large, even though I have already installed the Hadoop on our cluster, but HDFS is not a good choice for those “lots of small” files, I need to re-architect the cluster. MogileFS which is developed by Danga Interactive (This company is the creator of LiveJournal) and successfully utilized by Last.FM, can satisfy the requirements very well.

    Installation
    Before I can install MogileFS onto the cluster, I need first to prepare some computer nodes for it. Our cluster has more than 70 nodes, and I’ll use 6 nodes to setup the MogileFS, so, first I need to recommission the 6 nodes from Hadoop.

    According to one thread from ServerFault, First, we need to add the following lines to conf/mapred-site.xml, 
    <property>

    <name>dfs.hosts.exclude</name>
    <value>/etc/hadoop/conf.dist/dfs.hosts.exclude</value>
    <final>true</final>
    </property>
    where in the value part, we put the name of the file containing the 6 nodes. After this, usehadoop dfsadmin -refreshNodes to notify HDFS to transfer the data from the 6 nodes and delete the 6 nodes from the system. We can monitor this process from the webpage http://hadoop_master_data_node/dfsnodelist.jsp?whatNodes=LIVE

    Our cluster is deployed on Ubuntu, according to the incomplete document of MogileFS, installing MogileFS on Ubuntu is not that complicated.

    Development

    Performance

     
c
Compose new post
j
Next post/Next comment
k
Previous post/Previous comment
r
Reply
e
Edit
o
Show/Hide comments
t
Go to top
l
Go to login
h
Show/Hide help
shift + esc
Cancel