Just my blog

Since yesterday I enabled CloudFlare’s CDN (the Free Plan) without the use of their DNS infra.

I simply ‘looked up’ with nslookup my entries on their DNS servers and update my zone-files accordingly. 🙂 

I performed some investigation on availability/healing of GlusterFS Volumes.

In a “lab” environment I tested two types:

1. Distributed Volume
2. Replicated Volume

With the tests I wanted to figure out a few questions:

1. What is the availability of files if one of the nodes dies/goes offline
2. How is will GlusterFS recover of the node comes back online

Please note, that in the tests, the nodes were pulled offline manual, while the client was not connected to the node that goes offline. For a failover to a working node, you can use tools like ucarp for “failing over”.

Replicated Volume

Created a simple replicated volume across 3 nodes:

Then Node 1 dies/goes offline

All the files will be available for reading, since they are spread over all the nodes. Now you can also write data to the volume while node01 is offline.

Once Node01 comes back online, GlusterFS will “resync” the node:

Distributed Volume

We also created a simple distributed volume using 3 nodes:

And (again) node01 dies:

The files A and D are not available, since they are living on node01, the other files are still available. You can also write to the volume, but the files will be written on the nodes that are online (so not balanced across the 3 nodes):

Once Node01 is back online, the files A and D will become available again.

But, the volume is out of balance, so a rebalance should be initiated to get the replicated volume in a good shape:

As you can see there are pros and cons of replicated and distributed volumes, but you can also combine these:

But I haven’t test this set up in the lab… but can guess what will happen when a node die.

On one of the servers with GlusterFS, I use USB-Disks in a replicated GlusterFS volume. Although once in a while a disk just “dies” and using “replace brick” command is not an option.

So I figured out the following steps:

1. Stop the GlusterFS Daemon
2. Remove the faulty brick
3. Configure the new brick to mount on the same mountpoint as the old one
4. Start GlusterFS Daemon
5. Trigger a self-heal on the GlusterFS Volume
6. Get coffee/..
7. The new brick will contain the data

Please note, this only works for replicated volumes!

 

At home I set up a small 3 node GlusterFS Cluster based on CentOS 6.4 with one client with Fedora 16.

On the client and the nodes iptables and SELinux is disabled.

The GlusterFS package is installed using the EPEL repository.

Preparing the nodes

The nodes have been kick-started using the following %packages in the kickstart:

%packages
@core
@server-policy

After enabling the EPEL repository you run the following commands on all nodes:

yum -y install glusterfs glusterfs-server
mkdir -p /mnt/glusterfs/test/{balanced,replicated}
service glusterd start
chkconfig glusterd on

Now on one of the nodes we bind all the systems to the “cluster”:

[root@node01 ~]# gluster peer status
No peers present
[root@node01 ~]# gluster peer probe node01
Probe on localhost not needed
[root@node01 ~]# gluster peer probe node02
Probe successful
[root@node01 ~]# gluster peer probe node03
Probe successful
[root@node01 ~]# gluster peer status
Number of Peers: 2

Hostname: node02
Uuid: 6d5d5101-5a8e-4057-a854-2aac49ce22ca
State: Peer in Cluster (Connected)

Hostname: node03
Uuid: f651c972-7519-4b74-b610-f1f2f4db31ef
State: Peer in Cluster (Connected)

Create the volumes

We create two volumes, one replicated and one distributed.

Replicated volume

root@node01 ~]# gluster volume create VolReplica replica 3 transport tcp
node01:/mnt/glusterfs/test/replicated
node02:/mnt/glusterfs/test/replicated node03:/mnt/glusterfs/test/replicated
Creation of volume VolReplica has been successful. Please start the volume to access data.
[root@node01 ~]# gluster volume start VolReplica
Starting volume VolReplica has been successful
[root@node01 ~]# gluster volume info VolReplica

Volume Name: VolReplica
Type: Replicate
Status: Started
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: node01:/mnt/glusterfs/test/replicated
Brick2: node02:/mnt/glusterfs/test/replicated
Brick3: node03:/mnt/glusterfs/test/replicated

Balanced (distributed) volume

[root@node01 ~]# gluster volume create VolBalanced  transport tcp
node01:/mnt/glusterfs/test/balanced
node02:/mnt/glusterfs/test/balanced node03:/mnt/glusterfs/test/balanced
[root@node01 ~]# gluster volume start VolBalanced
[root@node01 ~]# gluster volume info VolBalanced

Volume Name: VolBalanced
Type: Distribute
Status: Started
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: node01:/mnt/glusterfs/test/balanced
Brick2: node02:/mnt/glusterfs/test/balanced
Brick3: node03:/mnt/glusterfs/test/balanced

Configure the client(s)

You can easily use one of the nodes also as client, but I nominated my (old) laptop with FC 16 as GlusterFS client.

First ensure the EPEL repositories are enabled, and then install the packages:

[root@laptoppie ~]# yum -y install glusterfs-fuse

create the (target) mount points:

[root@laptoppie ~]# mkdir -p /mnt/glusterclient/{replicated,balanced}

And mount the filesystems:

[root@laptoppie ~]# mount -t glusterfs node01:VolReplica /mnt/glusterclient/replicated
[root@laptoppie ~]# df -h /mnt/glusterclient/replicated
Filesystem         Size  Used Avail Use% Mounted on
node01:VolReplica   37G  924M   35G   3% /mnt/glusterclient/replicated
[root@laptoppie ~]# mount -t glusterfs node01:VolBalanced /mnt/glusterclient/balanced/
[root@laptoppie ~]# df -h /mnt/glusterclient/balanced/
Filesystem          Size  Used Avail Use% Mounted on
node01:VolBalanced  111G  2.8G  103G   3% /mnt/glusterclient/balanced

 

A long time I was enthusiastic about the fact that with Lotus Notes you were able to disable “reply-to-all” in a mail. After some search on Google, I noticed, that is also possible with MS Outlook. Scott Hanselman wrote a nice article on his weblog:

http://www.hanselman.com/blog/HowToEasilyDisableReplyToAllAndForwardInOutlook.aspx

To make it more user friendly, I modified the code a little bit:

Sub NoReplyAll()
‘——————————————————————–
‘Disable reply to all/forward options from a mail
‘This code is taken from
‘
‘ http://www.hanselman.com/blog/HowToEasilyDisableReplyToAllAndForwardInOutlook.aspx
‘
‘and some small modifications are applied by Pieter de Rijk
‘
‘ http://blog.adslweb.net
‘——————————————————————–

If ActiveInspector.CurrentItem.Actions(“Reply to All”).Enabled = True Then

‘Check the status if Reply-to-All is enabled, so we disable it
ActiveInspector.CurrentItem.Actions(“Reply to All”).Enabled = False
ActiveInspector.CurrentItem.Actions(“Forward”).Enabled = False

‘Give a prompt that it is disabled
MsgBox “Reply to all/Forward is disabled”, vbInformation, “Reply to all/Forward”
Else
‘The Reply-to-All is disabled, so enable it again
ActiveInspector.CurrentItem.Actions(“Reply to All”).Enabled = True
ActiveInspector.CurrentItem.Actions(“Forward”).Enabled = True
‘Prompt that we enabled it again
MsgBox “Reply to all/Forward is enabled”, vbExclamation, “Reply to all/Forward”
End If

End Sub

To offload my RaspBerry Pi with RaspBMC, I move away from the internal SQLite database to my MySQL server (as described on the XBMC Wiki).

After adding new sources and scraping the content… nothing showed up in the Movies overview… while the tables were populated…

In the xbmc.log file I found the following clue:

21:06:59 T:2928669760   ERROR: SQL: The table does not exist
                                      Query: select * from movieview
21:06:59 T:2928669760   ERROR: GetMoviesByWhere failed

 And after investigation, I noticed that there were no views defined in the schema.

To do a quick analysis on the sqllite database, I installed the SQLite Manager plugin for Firefox and found 4 views… and there I found the create statement for these views… so I simply copy-and-paste it into mysql… and it worked…

Unfortunately I was not able to insert the triggers that way… still have to find it out..

According to Bug ticket #13959 on the XBMC-Trac site, this was caused because I was a MySQL version < 5.16…. which was true… it was 5.0.something, so I upgraded my MySQL to version 5.1.67, and guess what… that did not work either…

So something goes “weird”, but looking into the source code didn’t gave me a clue so far… so up for further investigation…

---

The code to create the views:

CREATE VIEW episodeview AS SELECT   episode.*,  files.strFileName AS
strFileName,  path.strPath AS strPath,  files.playCount AS playCount, 
files.lastPlayed AS lastPlayed,  files.dateAdded AS dateAdded, 
tvshow.c00 AS strTitle,  tvshow.c14 AS strStudio,  tvshow.c05 AS
premiered,  tvshow.c13 AS mpaa,  tvshow.c16 AS strShowPath,  
bookmark.timeInSeconds AS resumeTimeInSeconds,  
bookmark.totalTimeInSeconds AS totalTimeInSeconds,   seasons.idSeason AS
idSeason FROM episode  JOIN files ON    files.idFile=episode.idFile 
JOIN tvshow ON    tvshow.idShow=episode.idShow  LEFT JOIN seasons ON   
seasons.idShow=episode.idShow AND seasons.season=episode.c12  JOIN path
ON    files.idPath=path.idPath  LEFT JOIN bookmark ON   
bookmark.idFile=episode.idFile AND bookmark.type=1;

CREATE VIEW
movieview AS SELECT  movie.*,  sets.strSet AS strSet,  files.strFileName
AS strFileName,  path.strPath AS strPath,  files.playCount AS
playCount,  files.lastPlayed AS lastPlayed,   files.dateAdded AS
dateAdded,   bookmark.timeInSeconds AS resumeTimeInSeconds,  
bookmark.totalTimeInSeconds AS totalTimeInSeconds FROM movie  LEFT JOIN
sets ON    sets.idSet = movie.idSet  JOIN files ON   
files.idFile=movie.idFile  JOIN path ON    path.idPath=files.idPath 
LEFT JOIN bookmark ON    bookmark.idFile=movie.idFile AND
bookmark.type=1;

CREATE VIEW movieview AS SELECT  movie.*, 
sets.strSet AS strSet,  files.strFileName AS strFileName,  path.strPath
AS strPath,  files.playCount AS playCount,  files.lastPlayed AS
lastPlayed,   files.dateAdded AS dateAdded,   bookmark.timeInSeconds AS
resumeTimeInSeconds,   bookmark.totalTimeInSeconds AS totalTimeInSeconds
FROM movie  LEFT JOIN sets ON    sets.idSet = movie.idSet  JOIN files
ON    files.idFile=movie.idFile  JOIN path ON   
path.idPath=files.idPath  LEFT JOIN bookmark ON   
bookmark.idFile=movie.idFile AND bookmark.type=1;

CREATE VIEW
tvshowview AS SELECT   tvshow.*,  path.strPath AS strPath, 
path.dateAdded AS dateAdded,  MAX(files.lastPlayed) AS lastPlayed, 
NULLIF(COUNT(episode.c12), 0) AS totalCount,  COUNT(files.playCount) AS
watchedcount,  NULLIF(COUNT(DISTINCT(episode.c12)), 0) AS totalSeasons
FROM tvshow  LEFT JOIN tvshowlinkpath ON   
tvshowlinkpath.idShow=tvshow.idShow  LEFT JOIN path ON   
path.idPath=tvshowlinkpath.idPath  LEFT JOIN episode ON   
episode.idShow=tvshow.idShow  LEFT JOIN files ON   
files.idFile=episode.idFile GROUP BY tvshow.idShow;

As mentioned before, I am playing around with Splunk’s authentication scripts with Centrify DC.
While playing I ran into a performance issue with the AuthenticationScripts implementation of getUsers.
According to the documentation the output is…

Recently I have been playing with Splunk’s Authenticationscript to retrieve information via Centrify DC, alhtough I noticed that an item for a user which is available in the webinterface for new users, is not available for users “created” via an authentication script.

If you read the specification the output of getUserInfo is:

--status=success|fail --userInfo=<userId>;<username>;<realname>;<roles> 

So for example:

--status=success|fail --userInfo=;pieter;Pieter de Rijk;admin

While in the webinterface you can also enter an e-mail address, so I raised a request with Splunk to get the e-mail address also populated via the authenticationscript-feature and it is picked up as an enhancement request.

I proposed the following output, so the authenticationscript stays backwards compatible:

--status=success|fail --userInfo=<userId>;<username>;<realname>:<mail-addres>;<roles>

So some output like (no comma, because that is used within a lot of companies with the DisplayName attribute like: “Rijk, Pieter de”):

--status=success|fail --userInfo=;pieter;Pieter de Rijk:[email protected];admin

I currently have a GlusterFS setup where the bricks are replicated, although brick 8 was having bad-sector issues, so should get replaced asap. After some searching I did not find how to do it… but is it pretty simple…

 gluster volume replace-brick GlusterVolume node1:/usr/glusterfs/brick_1_8 node1:/usr/glusterfs/brick_1_13 start

Depending on the speed of the disks (2TB USB disks in my case) it takes some while, you can monitor the status via:

gluster volume replace-brick GlusterVolume node1:/usr/glusterfs/brick_1_8 node1:/usr/glusterfs/brick_1_13 status

Once the replacing is “done” you simply do a commit to finalize the replacing:

gluster volume replace-brick GlusterVolume node1:/usr/glusterfs/brick_1_8 node1:/usr/glusterfs/brick_1_13 commit

That’s all