GlusterFS availability/healing of a Volume


I performed some investigation on availability/healing of GlusterFS Volumes.

In a “lab” environment I tested two types:

  1. Distributed Volume
  2. Replicated Volume

With the tests I wanted to figure out a few questions:

  1. What is the availability of files if one of the nodes dies/goes offline
  2. How is will GlusterFS recover of the node comes back online

Please note, that in the tests, the nodes were pulled offline manual, while the client was not connected to the node that goes offline. For a failover to a working node, you can use tools like ucarp for “failing over”.

    Replicated Volume

    Created a simple replicated volume across 3 nodes:

    Then Node 1 dies/goes offline

    All the files will be available for reading, since they are spread over all the nodes. Now you can also write data to the volume while node01 is offline.

    Once Node01 comes back online, GlusterFS will “resync” the node:

    Distributed Volume

    We also created a simple distributed volume using 3 nodes:

    And (again) node01 dies:

    The files A and D are not available, since they are living on node01, the other files are still available. You can also write to the volume, but the files will be written on the nodes that are online (so not balanced across the 3 nodes):

    Once Node01 is back online, the files A and D will become available again.

    But, the volume is out of balance, so a rebalance should be initiated to get the replicated volume in a good shape:

    As you can see there are pros and cons of replicated and distributed volumes, but you can also combine these:

    But I haven’t test this set up in the lab… but can guess what will happen when a node die. :-D

    Read More

    Replace a failing brick in a replicated GlusterFS Volume


    On one of the servers with GlusterFS, I use USB-Disks in a replicated GlusterFS volume. Although once in a while a disk just “dies” and using “replace brick” command is not an option.

    So I figured out the following steps:

    1. Stop the GlusterFS Daemon
    2. Remove the faulty brick
    3. Configure the new brick to mount on the same mountpoint as the old one
    4. Start GlusterFS Daemon
    5. Trigger a self-heal on the GlusterFS Volume
    6. Get coffee/..
    7. The new brick will contain the data :-)

    Please note, this only works for replicated volumes!

     

    Read More

    GlusterFS 10 minute guide on CentOS/RHEL/Fedora


    At home I set up a small 3 node GlusterFS Cluster based on CentOS 6.4 with one client with Fedora 16.

    On the client and the nodes iptables and SELinux is disabled.

    The GlusterFS package is installed using the EPEL repository.

    Preparing the nodes

    The nodes have been kick-started using the following %packages in the kickstart:

    %packages
    @core
    @server-policy

    After enabling the EPEL repository you run the following commands on all nodes:

    yum -y install glusterfs glusterfs-server
    mkdir -p /mnt/glusterfs/test/{balanced,replicated}
    service glusterd start
    chkconfig glusterd on

    Now on one of the nodes we bind all the systems to the “cluster”:

    [[email protected] ~]# gluster peer status
    No peers present
    [[email protected] ~]# gluster peer probe node01
    Probe on localhost not needed
    [[email protected] ~]# gluster peer probe node02
    Probe successful
    [[email protected] ~]# gluster peer probe node03
    Probe successful
    [[email protected] ~]# gluster peer status
    Number of Peers: 2

    Hostname: node02
    Uuid: 6d5d5101-5a8e-4057-a854-2aac49ce22ca
    State: Peer in Cluster (Connected)

    Hostname: node03
    Uuid: f651c972-7519-4b74-b610-f1f2f4db31ef
    State: Peer in Cluster (Connected)

    Create the volumes

    We create two volumes, one replicated and one distributed.

    Replicated volume

    [email protected] ~]# gluster volume create VolReplica replica 3 transport tcp
    node01:/mnt/glusterfs/test/replicated
    node02:/mnt/glusterfs/test/replicated node03:/mnt/glusterfs/test/replicated

    Creation of volume VolReplica has been successful. Please start the volume to access data.
    [[email protected] ~]# gluster volume start VolReplica
    Starting volume VolReplica has been successful
    [[email protected] ~]# gluster volume info VolReplica

    Volume Name: VolReplica
    Type: Replicate
    Status: Started
    Number of Bricks: 3
    Transport-type: tcp
    Bricks:
    Brick1: node01:/mnt/glusterfs/test/replicated
    Brick2: node02:/mnt/glusterfs/test/replicated
    Brick3: node03:/mnt/glusterfs/test/replicated

    Balanced (distributed) volume

    [[email protected] ~]# gluster volume create VolBalanced  transport tcp
    node01:/mnt/glusterfs/test/balanced
    node02:/mnt/glusterfs/test/balanced node03:/mnt/glusterfs/test/balanced

    [[email protected] ~]# gluster volume start VolBalanced
    [[email protected] ~]# gluster volume info VolBalanced

    Volume Name: VolBalanced
    Type: Distribute
    Status: Started
    Number of Bricks: 3
    Transport-type: tcp
    Bricks:
    Brick1: node01:/mnt/glusterfs/test/balanced
    Brick2: node02:/mnt/glusterfs/test/balanced
    Brick3: node03:/mnt/glusterfs/test/balanced

    Configure the client(s)

    You can easily use one of the nodes also as client, but I nominated my (old) laptop with FC 16 as GlusterFS client.

    First ensure the EPEL repositories are enabled, and then install the packages:

    [[email protected] ~]# yum -y install glusterfs-fuse

    create the (target) mount points:

    [[email protected] ~]# mkdir -p /mnt/glusterclient/{replicated,balanced}

    And mount the filesystems:

    [[email protected] ~]# mount -t glusterfs node01:VolReplica /mnt/glusterclient/replicated
    [[email protected] ~]# df -h /mnt/glusterclient/replicated
    Filesystem         Size  Used Avail Use% Mounted on
    node01:VolReplica   37G  924M   35G   3% /mnt/glusterclient/replicated
    [[email protected] ~]# mount -t glusterfs node01:VolBalanced /mnt/glusterclient/balanced/
    [[email protected] ~]# df -h /mnt/glusterclient/balanced/
    Filesystem          Size  Used Avail Use% Mounted on
    node01:VolBalanced  111G  2.8G  103G   3% /mnt/glusterclient/balanced

     

    Read More

    Outlook Disable “reply-to-all/forward”


    A long time I was enthusiastic about the fact that with Lotus Notes you were able to disable “reply-to-all” in a mail. After some search on Google, I noticed, that is also possible with MS Outlook. Scott Hanselman wrote a nice article on his weblog:

    http://www.hanselman.com/blog/HowToEasilyDisableReplyToAllAndForwardInOutlook.aspx

    To make it more user friendly, I modified the code a little bit:

    Sub NoReplyAll()
    ‘——————————————————————–
    ‘Disable reply to all/forward options from a mail
    ‘This code is taken from

    ‘ http://www.hanselman.com/blog/HowToEasilyDisableReplyToAllAndForwardInOutlook.aspx

    ‘and some small modifications are applied by Pieter de Rijk

    ‘ http://blog.adslweb.net
    ‘——————————————————————–

    If ActiveInspector.CurrentItem.Actions(“Reply to All”).Enabled = True Then

    ‘Check the status if Reply-to-All is enabled, so we disable it
    ActiveInspector.CurrentItem.Actions(“Reply to All”).Enabled = False
    ActiveInspector.CurrentItem.Actions(“Forward”).Enabled = False

    ‘Give a prompt that it is disabled
    MsgBox “Reply to all/Forward is disabled”, vbInformation, “Reply to all/Forward”
    Else
    ‘The Reply-to-All is disabled, so enable it again
    ActiveInspector.CurrentItem.Actions(“Reply to All”).Enabled = True
    ActiveInspector.CurrentItem.Actions(“Forward”).Enabled = True
    ‘Prompt that we enabled it again
    MsgBox “Reply to all/Forward is enabled”, vbExclamation, “Reply to all/Forward”
    End If

    End Sub

    Read More

    RaspBMC not creating view/triggers on MySQL


    To offload my RaspBerry Pi with RaspBMC, I move away from the internal SQLite database to my MySQL server (as described on the XBMC Wiki).

    After adding new sources and scraping the content… nothing showed up in the Movies overview… while the tables were populated…

    In the xbmc.log file I found the following clue:

    21:06:59 T:2928669760   ERROR: SQL: The table does not exist
                                          Query: select * from movieview
    21:06:59 T:2928669760   ERROR: GetMoviesByWhere failed

     And after investigation, I noticed that there were no views defined in the schema.

    To do a quick analysis on the sqllite database, I installed the SQLite Manager plugin for Firefox and found 4 views… and there I found the create statement for these views… so I simply copy-and-paste it into mysql… and it worked…

    Unfortunately I was not able to insert the triggers that way… still have to find it out..

    According to Bug ticket #13959 on the XBMC-Trac site, this was caused because I was a MySQL version < 5.16…. which was true… it was 5.0.something, so I upgraded my MySQL to version 5.1.67, and guess what… that did not work either…

    So something goes “weird”, but looking into the source code didn’t gave me a clue so far… so up for further investigation…


    The code to create the views:

    CREATE VIEW episodeview AS SELECT   episode.*,  files.strFileName AS
    strFileName,  path.strPath AS strPath,  files.playCount AS playCount, 
    files.lastPlayed AS lastPlayed,  files.dateAdded AS dateAdded, 
    tvshow.c00 AS strTitle,  tvshow.c14 AS strStudio,  tvshow.c05 AS
    premiered,  tvshow.c13 AS mpaa,  tvshow.c16 AS strShowPath,  
    bookmark.timeInSeconds AS resumeTimeInSeconds,  
    bookmark.totalTimeInSeconds AS totalTimeInSeconds,   seasons.idSeason AS
    idSeason FROM episode  JOIN files ON    files.idFile=episode.idFile 
    JOIN tvshow ON    tvshow.idShow=episode.idShow  LEFT JOIN seasons ON   
    seasons.idShow=episode.idShow AND seasons.season=episode.c12  JOIN path
    ON    files.idPath=path.idPath  LEFT JOIN bookmark ON   
    bookmark.idFile=episode.idFile AND bookmark.type=1;

    CREATE VIEW
    movieview AS SELECT  movie.*,  sets.strSet AS strSet,  files.strFileName
    AS strFileName,  path.strPath AS strPath,  files.playCount AS
    playCount,  files.lastPlayed AS lastPlayed,   files.dateAdded AS
    dateAdded,   bookmark.timeInSeconds AS resumeTimeInSeconds,  
    bookmark.totalTimeInSeconds AS totalTimeInSeconds FROM movie  LEFT JOIN
    sets ON    sets.idSet = movie.idSet  JOIN files ON   
    files.idFile=movie.idFile  JOIN path ON    path.idPath=files.idPath 
    LEFT JOIN bookmark ON    bookmark.idFile=movie.idFile AND
    bookmark.type=1;

    CREATE VIEW movieview AS SELECT  movie.*, 
    sets.strSet AS strSet,  files.strFileName AS strFileName,  path.strPath
    AS strPath,  files.playCount AS playCount,  files.lastPlayed AS
    lastPlayed,   files.dateAdded AS dateAdded,   bookmark.timeInSeconds AS
    resumeTimeInSeconds,   bookmark.totalTimeInSeconds AS totalTimeInSeconds
    FROM movie  LEFT JOIN sets ON    sets.idSet = movie.idSet  JOIN files
    ON    files.idFile=movie.idFile  JOIN path ON   
    path.idPath=files.idPath  LEFT JOIN bookmark ON   
    bookmark.idFile=movie.idFile AND bookmark.type=1;

    CREATE VIEW
    tvshowview AS SELECT   tvshow.*,  path.strPath AS strPath, 
    path.dateAdded AS dateAdded,  MAX(files.lastPlayed) AS lastPlayed, 
    NULLIF(COUNT(episode.c12), 0) AS totalCount,  COUNT(files.playCount) AS
    watchedcount,  NULLIF(COUNT(DISTINCT(episode.c12)), 0) AS totalSeasons
    FROM tvshow  LEFT JOIN tvshowlinkpath ON   
    tvshowlinkpath.idShow=tvshow.idShow  LEFT JOIN path ON   
    path.idPath=tvshowlinkpath.idPath  LEFT JOIN episode ON   
    episode.idShow=tvshow.idShow  LEFT JOIN files ON   
    files.idFile=episode.idFile GROUP BY tvshow.idShow;

    Read More

    Splunk authentication scripts – performance issue


    As mentioned before, I am playing around with Splunk’s authentication scripts with Centrify DC.
    While playing I ran into a performance issue with the AuthenticationScripts implementation of getUsers.
    According to the documentation the output is…

    Read More

    Splunk authentication scripts – missing the e-mail field


    Recently I have been playing with Splunk’s Authenticationscript to retrieve information via Centrify DC, alhtough I noticed that an item for a user which is available in the webinterface for new users, is not available for users “created” via an authentication script.

    If you read the specification the output of getUserInfo is:

    --status=success|fail --userInfo=<userId>;<username>;<realname>;<roles> 

    So for example:

    --status=success|fail --userInfo=;pieter;Pieter de Rijk;admin

    While in the webinterface you can also enter an e-mail address, so I raised a request with Splunk to get the e-mail address also populated via the authenticationscript-feature and it is picked up as an enhancement request. :-)

    I proposed the following output, so the authenticationscript stays backwards compatible:

    --status=success|fail --userInfo=<userId>;<username>;<realname>:<mail-addres>;<roles>

    So some output like (no comma, because that is used within a lot of companies with the DisplayName attribute like: “Rijk, Pieter de”):

    --status=success|fail --userInfo=;pieter;Pieter de Rijk:[email protected];admin

    Read More

    Gluster replace a brick that is replicated


    I currently have a GlusterFS setup where the bricks are replicated, although brick 8 was having bad-sector issues, so should get replaced asap. After some searching I did not find how to do it… but is it pretty simple…

     gluster volume replace-brick GlusterVolume node1:/usr/glusterfs/brick_1_8 node1:/usr/glusterfs/brick_1_13 start

    Depending on the speed of the disks (2TB USB disks in my case) it takes some while, you can monitor the status via:

    gluster volume replace-brick GlusterVolume node1:/usr/glusterfs/brick_1_8 node1:/usr/glusterfs/brick_1_13 status

    Once the replacing is “done” you simply do a commit to finalize the replacing:

    gluster volume replace-brick GlusterVolume node1:/usr/glusterfs/brick_1_8 node1:/usr/glusterfs/brick_1_13 commit

    That’s all :-)

    Read More

    YUM repo fails with koji/mock when base URL is used


    Recently I started playing around with Koji for package building.

    Everything was set-up pretty fast… but the first attempts to build a build-root failed…

    After some troubleshooting I found the cause…

    If you use the repo-data as available on an installation ISO (served via a webserver) and use mergerepos the location of an RPM will look like:

    <location xml:base="CentOS" href="pam-0.99.6.2-6.el5_5.2.x86_64.rpm"/>

    The entry as available on the ISO is:

    <location xml:base="media://1330913492.861127#1" href="CentOS/pam-0.99.6.2-6.el5_5.2.x86_64.rpm"/>

    The way to workaround this issue, is be recreated the repodata using createrepo:

    # createrepo -u http://172.16.3.240/repo/centos/5.8/base/x86_64/-o ${WEBSERVERPATH}/new-repo/ /media/cd/

    And then the entry will look like:

    <location xml:base=”http://172.16.3.240/repo/centos/5.8/base/x86_64/CentOS” href=”pam-0.99.6.2-6.el5_5.2.x86_64.rpm”/>

    The bug can be found in bugzilla.

    Read More

    SSL Chain


    I’ve ordered via http://www.cheapssls.com/ a simple SSL-certificate signed by Comodo for use with apache… although a lot of browsers (Firefox on Mac OS X, all browsers in Linux) didn’t accepted it (CA was not know…)
    After some discussion with …

    Read More

    ← OLDER POSTS NEWER POSTS →