Raspberry Pi 2B RAID 5 build – Part 3

Alright lets continue with our stock Raspberry Pi – Jessie build. It’s a long post, hang on to the handle bars!

First, install mdadm using the following commands:

sudo apt-get update #Updates our repository list
sudo apt-get install mdadm #
Accept the prompt during the installation.

Second, we’ll assume the disks are intended for a built from scratch RAID 5 array, so we’ll destroy all data on the disk (technically all we’re doing is remove previous partition table and formatting information, it could still be recovered  with data recovery software, many available on the Internet).

We’ll also assume 3 disks. More can be used, but not less than 3 for a RAID 5 array, that’s a requirement. In my case I have 3 powered external USB disks directly plugged in to the Pi.

Third, identify the disks using lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT 

lsblk

Here are my three disks, sda1, sdb1 and sdc1 (a through c denoting the sequence of the disks). My disks are partitioned already in the above picture (we’ll talk about that below) so they have numeric sequencing for their disks (the number 1 as all use the first partition). If they were never partitioned then you’ll see sda, sdb, sdc and need to partition them before use.

Be very careful and verify that the disks identified are indeed disks that you intend to use! Once we proceed all previous data will be erased!

Fourth, we’ll start by partioning them using the command sudo parted. Once launched you’ll see a prompt with (parted). We’ll enter the following commands at this prompt. Start creating the partitions.

select /dev/sda
mklabel gpt
mkpart primary ext4 1049KB 3TB

select /dev/sda selects the first disk to be partitioned.
mklabel creates a partition table of type gpt (specifically needed for Linux and large multi Terabyte disks).
mkpart creates the partition. In this case I am starting at 1049KB (never mind the reasoning …) and ending at 3TB as that is the size of the disks I have (a 3TB external USB drive).

Proceed to the other disks (sdb and sdc replacing sda) the same way.
Enter quit to exit out of parted.

Fifth, we create the RAID 5 array using the following commands. This will prevent from the array created with spares (which we don’t want, we want to use all the disks for the maximum capacity we can get, we’re already losing 1/3 of the total disk space for parity information).

sudo mdadm –create –verbose –force –assume-clean /dev/md0 –level=5 \
–raid-devices=3 /dev/sda1 /dev/sdb1 /dev/sdc1

Check that they’re running with the following command

cat /proc/mdstat

You should see:

mdstat

Lets create the filesystem (this is what most resembles the Format command in Windows).

sudo mkfs.ext4 -F /dev/md0

You should see:

mkfs

It is now ready to mount so we can access it

sudo mkdir -p /mnt/raid5

You can now mount it using

sudo mount /dev/md0 /mnt/raid5

and see it’s content by

ls -al /mnt/raid5

mount

You can also confirm its capacity by (note /dev/md0 with 5.5T of size).

df -h -x devtmpfs -x tmpfs

mount

We’ll save the configuration in /etc/mdadm/mdadm.conf so it starts up at boot time automatically.

sudo mdadm –detail –scan | sudo tee -a /etc/mdadm/mdadm.conf

Update the initial file system (Raspberry Pi uses a RAM disk image when booting up and we want to include our array).

sudo update-initramfs -u

We’ll add the drive array to our filesystem table also so it will be mounted automatically when booting up.

echo ‘/dev/md0 /mnt/raid5 ext4 defaults,nofail,discard 0 0’ | sudo tee -a /etc/fstab

Finally confirm the array gets mounted by rebooting the Pi.

sudo reboot

You should see the following when it has booted.

ls

If “lost+found” entry isn’t listed there is more work to do (which I had to do with my Pi).

Edit /etc/rc.local file

sudo pico /etc/rc.local

Add the following line right before the line containing “exit 0”

/usr/local/bin/raid5 &

Then we create a shell script at /usr/local/bin/raid5

sudo pico /usr/local/bin/raid5

Enter the following lines in it

#!/bin/bash
# Sleep for 20 seconds after boot to make all the disks available for array assembly
sleep 20
sudo mdadm –assemble –scan # Assemble the RAID 5 array

Then set the execute permissions for the file

sudo chmod +x /usr/local/bin/raid5

Now reboot and the raid array should mount automatically. If it still doesn’t then add the following to /usr/local/bin/raid5

mount -a

Once mounted run the following command so I can start populating the /mnt/raid5 folder.

sudo chmod ug+wrx /mnt/raid5

That’s it!

Next I’ll setup samba (SMB Windows shares for Linux) and share out the folder so I can connect to it from other machines …

Next move on to part 4: https://caesarsamsi.wordpress.com/2017/01/03/raspberry-pi-2b-raid-5-build-part-4/

Raspberry Pi 2B RAID 5 build – Part 2

Now we’re ready to do the semi difficult parts.

Write the Raspbian Linux OS image to the micro SD card. I use a Mac so I use the following instructions: https://www.raspberrypi.org/…/inst…/installing-images/mac.md

Note that when you are in the Mac’s System Report window, expand Hardware -> USB -> USB2.0 Hub -> USB 2.0 Hub then click on USB Storage. Or whatever you do, find that USB Storage node. In the details panel below on the right you’ll find “BSD Name”, in my case it is disk5, that’s the micro SD card we’re writing the image to (the Pi instructions missed this very tiny important detail).

system-reporthardware

If you use Windows then use the following instructions:
https://www.raspberrypi.org/…/…/installing-images/windows.md

For Macs, when you use the sudo dd command to write the image to the micro SD card, a progress update won’t be displayed. Instead press Control-T to see a status update when you want.

The process takes about 4.5 minutes on my Mac. It’ll be roughly the same on other computers as the time is dependent on the USB 2.0 port and microSD speed. I have a Class 10 micro SD card, which is pretty fast at the time this is posted.

Once you’re done with writing the Raspbian Linux OS image, boot up the Pi.

It is already preconfigured to provide X Window for a GUI display, and the network is preconfigured to use DHCP.

It’s now a ready to use Linux desktop, though I like the term tinypute myself

Next we’ll install and build the RAID 5 array itself …

On to Part 3 of the journey https://caesarsamsi.wordpress.com/2017/01/03/raspberry-pi-2b-raid-build-part-3/

Raspberry Pi 2B RAID 5 build – Part 1

First, you need a Raspberry Pi 2B computer. You can purchase it anywhere. It’s home is here: https://www.raspberrypi.org/

Remember to purchase a micro USB cable with a power charger which is used as the Pi’s power supply, also remember to purchase an HDMI cable.

The HDMI cable is used to display the Pi’s video output on an HDTV monitor (1080p). I’ve only used the HDMI display once to find the IP address of the Pi, after that I typically use the terminal/ssh to connect and work with the Pi.

In case they don’t come with the USB 2.0 cable (they should), also remember to buy 4 USB 2.0 cables.

Second, you need 4 USB 2.0 external hard disks. The Pi has 4 USB 2.0 ports so up to 4 disks can be connected directly. You can use more disks but you’ll have to use a USB hub. Using a hub to use more than 4 USB disks will degrade performance as a single USB 2.0 connection is shared amongst many disks.

Third, you’ll need a micro SD card, 8GB will do. This will be the Pi’s Linux OS hard disk.

Fourth, you’ll need to download the OS (it’s called Raspbian and is actually Linux in the Debian distribution form. Get it? Rasp(berry) and (De)bian .

You can download it from here: https://www.raspberrypi.org/downloads/raspbian/

Next we’ll write the image to the SD Card …

On to part 2 of the journey https://caesarsamsi.wordpress.com/2016/12/31/raspberry-pi-2b-raid-build-part-2/

Raspberry Pi 2B RAID 5 build – the Journey

This is my journey in building my own at home private cloud

It has no high availability from the processing node perspective as there is no fail-over/backup processing hardware to make this a highly available cluster. There is a solution to this as described in http://www.linuxjournal.com/content/two-pi-r?page=0,1 if you like the challenge. I might do that at another time.

This implementation only has disk high availability from 3 x 3TB USB 3.0 external disks turning this build into a nice NAS for my PCs, VMs, and laptops. We’ll see how slowly it crawls. It actually has decent performance for daily routines. Large file size & number copies will make it crawl. I just do that overnight (copied my iTunes library over).

Yes, I’m a digital pack-rat .

Go on part 1 of the journey! https://caesarsamsi.wordpress.com/2016/12/31/raspberry-pi-2b-raid-build-part-1-2/

pichristmas

Install a LAMP and mailman (mailing list) server

mailman

I needed a mail server for a few custom email addresses from several domains that I own. There are some 6 domains and 10 custom email addresses.

Multiple email domains can be hosted on a single mail server. They are called virtual domains. Basically email1@domain1.com and email2@domain2.com can be delivered to mailserverdomain.com. Clients from domain1 and domain2 connect for email service to mailserverdomain.com

This configuration moves my email addresses from outlook.office365.com which charges $5.- / month / email account. Totals to saving a sweet $50.- monthly. That’s $600.- a year which I could use for something else.

I begin by installing the Mail Server using this cookbook: LAMP + Mail Server + Web Mail Server then I install mailman using my own cookbook: mailman-2-1-23-install

This results in a Mail Server capable of hosting multiple domains with multiple emails accounts. In fact it is running in my home office on a Raspberry Pi box!

If you want something like this but skip about 200+ lines of instructions, let me know, I’d be happy to help!

 

 

Connecting from Mac to Raspberry Pi’s remote desktop using X11

I play a lot with my Raspberry Pi PC. Usually I connect using Mac’s Terminal application via ssh to the Pi.

Sometimes though, I need to use the Pi’s full graphical desktop. For example I’m much more comfortable partitioning disks using gparted vs. any other command line tool.

So for those occasions I use an X11 window to connect from the Mac to the Pi, I use ssh to connect, and then run lxsession as the desktop on the Pi.

First, install X11 from XQuartz (http://www.xquartz.org).

Download the installer package disk image from the homepage. Open the disk image and run the installer package. The installation will take a few minutes and towards the end it will say there’s 1 minute left for quite a while 🙂

Once the installation is complete you must log out and log back in. If you don’t, the remote desktop won’t open. XQuartz is installed to Applications -> Utilities

Second, launch XQuartz then open XQuartz’s Terminal application (XQuartz menu -> Applications -> Terminal).

Third, connect to the Pi using ssh inside the XQuart’z Terminal. Then Enter ssh -X user@pi (replace user with your username on Pi, and pi with the name of your Pi).

Fourth, launch the Pi’s desktop. Enter lxsession. The Pi’s desktop should now be running on the Mac’s desktop.

Cloning/Backing Up Raspberry Pi using a Mac

This grew out of the need to create multiple SD card Raspbian images for my Raspberry Pi Hadoop cluster. I was constantly experimenting and really wanted not to create the Raspbian OS from scratch every time (note that Raspbian OS is really Linux of the Debian variant).

I sought about how to create an image file of a Raspbian installation so I could flash the image to an SD card when I needed a fresh Raspbian OS installed.

A running Raspbian OS installation configured as necessary is required.That installation’s SD card is used to create the image and clones.

Cloning doesn’t really save time, the time taken to write the image a scratch SD card is the same as writing the clone image, but it saves on errors from human forgetfulness and typing. It also makes the base configuration easy as Pi (LOL), known and fixed.

First thing to know is the size of the source SD card matters. The disk image can only be flashed to an SD card of the same or larger size. I chose 16GB size as they are readily available for a reasonable price.

First, I use Mac’s Disk Utility tool. Applications -> Utility-> Disk Utility app.

Second, I insert the micro SD card with a Raspbian OS installation into the micro SD card reader and then insert the reader in the Mac’s USB port. A new disk shows up in Disk Utility.

Third, I click on the disk and then click on the “New Image” item on the menu bar on top, choosing “DVD/CD Master”as the Image Format and click “Create”. I use cloneimage.cdr as the filename, this will actually create an .ISO file, it will create the image file to be used later to clone an installation.

Whenever I need a new Raspbian installation I do the following.

First, I insert a new micro SD card in the card reader and insert the reader in your Mac’s USB port.

Second, I launch the Terminal utility application from Applications -> Utility. I enter “diskutil list” and make note of the disk name, this would be in the form of /dev/diskN where N is a number. I typically find the right disk by looking at the SIZE column and find my 16GB card that way.

Third, I unmount the disk using: “diskutil unmountdisk /dev/diskN”

Forth, this step overwrites the card in the reader. WARNING: this step will not pause for confirmation. If the wrong disk is specified, that disk will be overwritten, so I’m very careful and double check. Enter: “sudo dd if=cloneimage.cdr of=/dev/rdiskN bs=10m”

Fifth, well, I wait until the image has finished writing, there is no ongoing progress status output but pressing Ctl-T will print some status information.

About the only indication is the app finishes running and the console prompt shows again.

That’s it, the only way to know is to boot up the new Pi with the new card.

Cleanup, I usually clean up the new Pi by changing the hostname, if I picked an installation with a static IP address then I’ll update the static IP address. Depending on the installation, sometimes I also change users and passwords.