Backups, cheaper and faster?

When did backups become easy and affordable?
I've been sort of slacking off in the backup game in the last year or so, our backups have been managed by other parts of the company and some backups have been outsourced to BSP's. I recently spec and deploy a basic Netbackup solution for our office servers at work and was amazed how extremely simple the whole thing was. The first backup system I worked with was Legato Networker (now owned by EMC) and it was at the time a pretty great product but it was quite difficult to manage, lots of things needed to be scripted and setting up a new tape library required quite a lot of tweaking.
Secondly, the budget was at first glance fairly limited but when we speced up what we needed and got a couple of quotes in I was amazed of how much we actually got for our money.
I wanted basic disk staging then duplication to tape. The obvious choice for tape was LTO4, we get pretty good price / performance, market leading capacity and it's a solid standard.
Most suppliers actually quoted the same tape library, just branded by different vendors. HP, Dell, IBM. All where IBM 3573 tape libraries, it's a excellent and affordable library. Dell came in best in price as usual. We opted for SAS attached. The backup server, my first choice server these days, is the Dell PowerEdge 2950. I buy it for pretty much 75% of all requirements. Speced out with a pair of Quad-Core 1.86GHz processors, 4Gb RAM, 6x500Gb SATA drives (in RAID5) for the disk staging pool and a SAS5 to attach the library.
Software wise the server is running CentOS 5.1 and Symantec/Vertias Netbackup 6.5 “Linux Starter Edition (LSE)”. The LSE is a pretty good deal, you get a server license, tape library license and 5 client licenses (with bare metal, open file etc.) for a fairly low cost.
Installing and configuring Netbackup was as easy as it gets, the tape library was detected on boot, no sysctl.conf changes needed. Stepped through the netbackup installation process, added the tape library (less than 5 clicks!!), defined a couple of tape label policies, created a 2Tb disk staging pool with my new tape group as the target, just add water (ehm, clients actually).

The total cost was under £13.000 (about $22.000), installation time was less than one day, included was:

• Veritas Netbackup 6.5 Linux starter edition
  
• Netbackup clients 5-pack


• Netbackup Exchange agent


• Dell PowerEdge 2950 server


• Dell PowerVault TL2000 library w/ one LTO4 SAS drive


• 25x LTO4 tapes


• 3x LTO cleaning tapes


• CentOS Linux 5.1 (free)

Cyrus IMAP file system tuning

Been busy, not enough blogging, bla bla bla. I know.
Just a lot of stuff going on at work at the moment, mergers and integrations.

We've had some problems with one of our IMAP servers at work running Postfix and Cyrus IMAPd. A for the job quite well speced machine, dual Xeons and 3x146Gb disk in RAID5 (4x146 in RAID10 would have been nicer). Anyway, the machine has got 50 or so IMAP users and perhaps 100Gb spool data on a ReiserFS partition.
The machines' avg. load has peaked at over 6.00 with about 75% in iowait on a bad day. I suspect that modern fancy e-mail search tools are to blame for the problems, applications building search indexes and such (Apple Mail anyone?).
Monitoring of the server showed quite a lot of inode update activity, even though there isn't *that* much new email coming in.
Must be our old (not so) dear friend atime that's making a little mess, I've used the noatimea and nodiratime mount options in the past with great success. Seen performance improvements of a couple of percent.
The mount-options noatime and nodiratime simply disables the feature to update the access timestamp of a file (and directory). I.e. when someone clicks and reads an email in their mail application the inode atime timestamp is updated. When is this atime timestamp used? Never.
Did a quick online remount of the spool fs with noatime and nodiratime.
The result?
Avg. load hasn't touch 1.00 since. Wow! I was expecting an improvement, but not that big.
Great and easy way to improve performance on IMAP spools.

mount -o remount,noatime,nodiratime /var/spool/imap

And don't forget to update /etc/fstab with the same mount options.
Cool.

Oracle under CentOS 5

I'm a bit late on this one, but ...
CentOS version 5 was released the other day. For those of you unfamiliar with CentOS it is RedHat Enterprise Linux recompiled and supported by the open source community.
Pretty much RHEL for free.
I can't say there is anything *really* exciting in the new version of RedHat, sure there is Xen support and the general updated versions of the applications included.
New version of Open Office, PostgreSQL etc.

One big difference is that everything has to be NTPL threaded. NPTL threading is the first real *good* threading implementation in Linux. The old Linux threads model is way old. Although NPTL has been available (and the default) in RHEL since version 3, it is not the only supported threading model.

-So what does this mean for Oracle?
No more LD_ASSUME_KERNEL hacks.
10g is fine with this, no problem at all. However 9i will have some issues with this, the old way to get Oracle running was simply to set LD_ASSUME_KERNEL to 2.4.19 and Linux would use the Linux threads model. No go in version 5.
Big question, will Oracle support 9i under RHEL5? I'm honestly not to bothered, by the time RHEL5 is mature and tested I really hope that people have switched to 10gR2.
9iR2 has been around for a lng time now and will be for some time. But new system design implementation on 9iR2?

I've done a quick test install of Centos 5 and 10gR2 (10.2.0.1 and patchset 10.2.0.2) and the install was very painless, pretty similar to Centos 4 with a few changes in the RPM's needed.
RedHat did add a few SHM parameters to sysctl.conf so that section needs reviewing though.

bash globbing and dot-files

Found a nifty little feature in bash.

Globbing is expanding file pattterns, like when you typ "ls -l file*" in bash it is not ls that does the file matching and filtering. bash will glob ("expand") the file list file* and ls will get all the files as arguments.
Now, per default bash doesn't glob dot-files. If I do "ls *" I will not get .bashrc and .bash_profile etc. Luckily it is easy to change this behavior. Set the bash option dotglob to enabled and it works!
Example:

[hlinden@spinner testdir]$ ls -Al
total 0
-rw-rw-r--  1 hlinden hlinden 0 Feb  1  2007 .dotfile1
-rw-rw-r--  1 hlinden hlinden 0 Feb  1  2007 .dotfile2
-rw-rw-r--  1 hlinden hlinden 0 Feb  1  2007 file1
-rw-rw-r--  1 hlinden hlinden 0 Feb  1  2007 file2
-rw-rw-r--  1 hlinden hlinden 0 Feb  1  2007 file3
[hlinden@spinner testdir]$ echo *
file1 file2 file3
[hlinden@spinner testdir]$ shopt -s dotglob
[hlinden@spinner testdir]$ echo *
.dotfile1 .dotfile2 file1 file2 file3
[hlinden@spinner testdir]$

Just put shopt -s dotglob in your .bashrc file or in a global /etc/profile.d file.
Another quite nice globbing feature is to have case insensitive globbing.
Check this out:

[hlinden@spinner testdir]$ shopt -s nocaseglob
[hlinden@spinner testdir]$ ls -l F*1
-rw-rw-r--  1 hlinden hlinden 0 Feb  1  2007 file1
[hlinden@spinner testdir]$

Testing iSCSI

I've been reading a lot about iSCSI lately, Sun is pushing the ZFS and iSCSI integration in the latest release of Open Solaris and it's the new hot thing in the Oracle community. I remember testing the Linux target ("server side") a few years ago and back then it was quite flaky, far from production ready.
This time around it seems pretty solid.

I just used the very simple instrucions here to compile the iscsi kernel module in Ubuntu, I created two "zero files" using dd (one for my VMware Linux test and one for my Vista laptop) and added them as LUN's to the /etc/ietd.conf config file.

# cat /etc/ietd.conf
Target starbuck
        Lun 0 Path=/u03/starbuck_lun0.dat,Type=fileio
Target oravm
        Lun 0 Path=/u03/oravm_lun0.dat,Type=fileio

The installation in CentOS 4 was simple, do a yum -y install iscsi-initiator-tools and set the discovery host and initiator name in /etc/iscsi.conf. Starting /etc/init.d/iscsi gives:

iscsi-sfnet: Loading iscsi_sfnet version 4:0.1.11-3
iscsi-sfnet: Control device major number 254
iscsi-sfnet:host3: Session established
scsi3 : SFNet iSCSI driver
  Vendor: IET       Model: VIRTUAL-DISK      Rev: 0
  Type:   Direct-Access                      ANSI SCSI revision: 04
SCSI device sdd: 524288 512-byte hdwr sectors (268 MB)
SCSI device sdd: drive cache: write through
SCSI device sdd: 524288 512-byte hdwr sectors (268 MB)
SCSI device sdd: drive cache: write through
 sdd: unknown partition table
Attached scsi disk sdd at scsi3, channel 0, id 0, lun 0

Create a partition and file system on the whole disk with

printf "n\np\n1\n\n\nw\n" | fdisk /dev/sdd && mkfs.ext3 /dev/sdd1

Done!

Conneting the iSCSI disk in Vista was pretty straight forward as well, add the iSCSI discovery IP and do a refresh of your targets.

Read about Oracle RAC and iSCSI in this article.

Oracle ASM instance startup script

Tought I'd publish my Oracle ASM instance sysv (Redhat/CentOS/Unbreakable etc) init script. The default oracleasm script provided by asmlib only prepares the disks available to ASM, you also need to start the basic instance for ASM. I've set the ASM instance to start just after oracleasm and to be killed just before it. Make sure your normal Oracle instance startup scripts are started in the correct order, i.e. after this script for startup and before this script for shutdown.
It reads the default /etc/sysconfig/oracle config file, if nothing it sets it goes to defaults and should be able to start the instance. If you don't have the ASM instance name set in /etc/sysconfig/oracle (as ASM_SID=FOO) and your instance name is not +ASM you may have to edit the script to change the default ASM SID.

Download the script here. Copy the script to /etc/init.d/ and use chkconfig --add oraasm to add it to startup. Do a chkconfig --list oraasm to verify that the script was added the runlevels you use.

Please report any bugs, problems or suggestions for the script to me.

rlwrap for rhel / centos / unbreakable

Just buil a new RPM file for rlwrap 0.26 for Centos 4, Redhat 4 and Oracle Unbreakable Linux. I've added my (very) basic sqlplus auto complete file.

If you are not familiar with rlwrap but use Oracle on Linux/Unix you should definatly have a look. rlwrap provides basic "bash style" command line editing in sqlplus (and other command line tools), you get command history with working arrow keys and normal command line editing.

Download 32-bit version: rlwrap-hali-0.26-1.i386.rpm
Download 64-bit version: rlwrap-hali-0.26-1.x86_64.rpm

Running Oracle Unbreakable Linux under VMWare

I finally got around to do a proper install of Unbreakable Linux today.
Pretty much as expect, it's pretty much identical to CentOS. It's got yum and all that.

Took the opertunity to try the new free version of VMWare GSX Server, or at least what used to be known as GSX. They just call it VMWare server now.
The VMware tgz install on my 64-bit Ubuntu 6.10 was quite smooth. You still have to enter a serial, easily obtained from filling out a simple form on the vmware website. The installation of vmware-authd failed though. Since Ubuntu's xinetd doesn't read /etc/inetd.conf I had to create an entry in /etc/xinet.d/.

hlinden@htpc:~$ cat /etc/xinetd.d/vmware
service vmware
{
        disable         = no
        socket_type     = stream
        protocol        = tcp
        user            = root
        wait            = no
        user            = root
        server          = /usr/sbin/vmware-authd
        port            = 902
}

Speed wise VMWare server is great, I only allocated 384Mb RAM to the VM and it still feels quite snappy and fast. I've only tried running two VM's at the same time so far. But my first impressions are simply, great stuff! The VM creation wizard thingy even had Solaris x86 in the list, haven't tried it yet but I sure will.

Took a few screenshots during the Unbreakable Linux installation.

Oracle Unbreakable Linux

Ok, so Oracle is doing "a CentOS".
Oracle is taking the Redhat Enterprise Linux source code, removing all the trademarks, logos and references to Redhat and recompiling it with the Unbreakable Linux label. If there are any bugs found and redhat don't fix them fast enough (at all?) Oracle will take care of that and guarantee all bugs will be fixed and backported to all versions of Unbreakable Linux.
Sounds pretty good to me, especially when you look at the support cost model.

The basic support contract "network level" gives you access to patches via Oracles up2date equivalent. And the cost is a mear $99 per year per system.
If you want enterprise support you get that for a couple of hundred more.
If you want premium support you get that for about $1399 per year, you can't even get that support level from Redhat.
The full support policy can be found here.
And... even better... support contract are 50% off for the first few months (although I still haven't found where you can actually buy them).



Read more. Take a few minutes (about 60) to watch Larrys keynote, quite interesting.

Download ISO images, free for all.

Oracle 10gR2 on Ubuntu 6.10 AMD64

I just finished installing Oracle 10gR2 on my Intel Core2 Ubuntu 6.10 machine running in x86_64 (AMD64) mode. The installation is fairly straight forward with a few gotchas. In comparison to CentOS/Redhat, Ubuntu 6.10 has got much more native 64-bit libraries, pretty much everything is actually 64-bit. Under Redhat most libraries are 32-bit. And the Oracle Universal Installer (OUI) uses a 32-bit Java runtime so hence we have to install a few i386 compatability libraries to get it working.

First of all make sure your machines hostname resolves to your external IP and not the loopback IP (127.0.0.1).
Double check using this command

$ grep $(hostname) /etc/hosts
192.168.95.40           htpc

If you see 127.0.0.1 just edit the hosts file to have the external IP.

Now you need to install a few compat libraries and 32-bit libraries.

sudo apt-get install gcc libaio1 lesstif2 
  lesstif2-dev make libc6 libstdc++5 \
  lib32stdc++6 lib32z1 ia32-libs

This will probably cascade in to quite a few packages depending on how much you have installed, be patient and let it download everything it needs. The extra ia32 libraries is probably the biggest change to the old 32-bit installation.

Edit /etc/sysctl.conf to have the kernel parameters required by Oracle (I've added a few extras here as well to avoid unecessery swapping).
Add these lines:

# Oracle stuff
kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 262144
net.core.rmem_max = 262144
net.core.wmem_default = 262144
net.core.wmem_max = 262144
vm.swappiness=10

Create the oracle user and required groups (you don't really need the nobody group, it's only used by root.sh to be the group owning the oraenv and dbhome scripts).

sudo groupadd nobody
sudo groupadd oinstall
sudo groupadd dba
sudo useradd -s /bin/bash -g oinstall -G dba oracle
sudo passwd oracle

Increase system limits by adding these lines to /etc/security/limits.conf:

* soft nproc 2047
* hard nproc 16384
* soft nofile 1024
* hard nofile 65536

To avoid the non-redhat warnings during the prereqs run this command:

echo 3 > /etc/redhat-release

Add a few symlinks to avoid any script problems:

sudo ln -s /usr/bin/awk /bin/awk
sudo ln -s /bin/true /bin/rpm
sudo ln -s /usr/bin/basename /bin/basename
sudo ln -s /lib/libgcc_s.so.1 /lib/libgcc_s.so
sudo mkdir -p /u01/oracle/10g
sudo chown -R oracle:oinstall /u01/oracle/10g
sudo chmod -R 775 /u01/oracle

Time to run the installer, make sure you are allowed to use the X11 server (export DISPLAY, xhost, vnc whatever).
To get the 32-bit JRE to work you need to set the XLOCALELIBDIR before running the installer:

export XLOCALELIBDIR=/usr/lib32/X11/locale
cd /your/install/dir
./runInstaller

I just ran the defaults straight through (except the SID name and password). The installation without any warnings or errors. The linker worked fine and everything was configure properly, dbca created the standard database.

oracle@htpc:~$ uname -a
Linux htpc 2.6.17-10-generic #2 SMP Fri Oct 13 15:34:39 UTC 2006 x86_64 GNU/Linux
oracle@htpc:~$ rsqlplus / as sysdba

SQL*Plus: Release 10.2.0.1.0 - Production on Sat Oct 21 20:13:45 2006

Copyright (c) 1982, 2005, Oracle.  All rights reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options

SQL> select platform_name from v$database;

PLATFORM_NAME
--------------------
Linux 64-bit for AMD

SQL>

This guide is based on the Oracle 10gR2 x64_64 installation guide and I've also used Dizwells excellent Ubuntu 6.06 32-bit guide for some ubuntu specific notes. Thanks.

Cool, now I can alt-tab between sqlplus and my latest Prison Break episodes on the TV . :)

Dell 9th generation servers

Quick overview of Dells 9th generations rack-mount server line up

It's been a couple of months now since Dell introduced the first machines in the new "9th generation" server line-up, the PowerEdge 1950, 2950, 2900 and the Blade 1955 all looked like quite nice machines, they can be fitted with the new DRAC/5 card and onboard PERC.
And they have recently been accompanied by slightly simpler (and more cost effective) SC1435 and the baby 860.

All machines use the new generation Intel processors (64-bit), FB-DIMM memory and SAS and/or SATA disk drives.
I'm not going to go on about products specs and other sales crap, you can find all that on the Dell website. Just my view on the machines and what they are suitable for. Everything from using Linux as the OS on the box. Solaris 10 is currently not confirmed as working, using 06/06 you still need a driver disk for the PERC card and ethernet is unsupported so you need a addon Intel PCI card to even make use of the server.

Dell PowerEdge 1950
Nice 1U pizza box machine. I would use this as an application or web server, if you equip it with the 4 2.5" disk backplane you could possibly have a very small database on it. Disk I/O is very likely to be a bottle neck though. Hooking it up to a SAN is another option. Good layout of the internal components and the back connectors are easy to access. The PCIe slots are a bit limited in lenght, but then again there aren't many long cards available any more (anyone remember the old full lenght PCI raid cards?). Quite redundant with dual "most things".
To summarize, a perfect box for a Apache HTTPd or Java app server.
Starting price in a usable spec: $3437


Dell PowerEdge 2950
Pretty much the same machine as the 1950 but with more I/O. Six 3.5" drives or eight 2.5" drives. Decent small office file server, takes 1.8Tb internally (3Tb if you go SATA), room for extra Ethernet cards if you want to run iSCSI. Would also be a good "all in all" intranet server, put Postgres, PHP, Postifx and few other things on it. Or install a Zimbra and use it as a e-mail server.
Slap 8 2.5" drives in it and it is a nice decent DB server. If you team up a group of these and hook them up to a SAN they make great RAC-nodes. As the 1950 it is fairly redundant and can be trusted to work by itself for non-critical things.
Starting price in a usable spec: $3757


Dell PowerEdge 2900
Not really a rack-mount box but it does come with a rackmount kit. Only included here as it is the only "one box solution" for larger installations such as a Oracle EE, mid-sized file server or even a large e-mail server. Takes up to 10 internal drives (8 + 2 cage), thats 3Tb with current SAS drives. Lots of PCIe I/O, six slots for HBAs and NICs. If you like virtualization it would make a great VMWare box, it takes an impressive 48Gb RAM (you don't want to see the price tag for this though).
5U in a rack though, I hope you're not paying per U if you colocate it.
Starting price in a usable spec: $4370


Dell PowerEdge 860
This one I don't really understand. First of all, who came up with the name 860, it is clearly a 950. Second of all I don't really get where this box is meant to be used, the SC1435 is a better choice in 90% of all cases. The only advantage over the SC1435 is the fact that it has the normal kick ass DRAC/5 card as an option. It takes Celeron, Pentium D and Xeon 3000 series processors, I really *really* recommend against using a Celeron proc in anything that goes in a rack.
Possible uses (if I have to) would be a DNS server, mail relay, basic web server or management server.
Starting price in a usable spec: $1391

Dell PowerEdge SC1435
This little guy is not officially out yet, should be available in the comming weeks. The tower equivalent SC1430 is currently available though. I must admit that this is probably one of my favorites in the line-up, not because it's a great flexible, highly redundant machine. But simply because it's cheap and gets the job done! No hot swap anything. This is probably the perfect app or web server for larger load balanced solutions. This is however an AMD Opteron Socket-F server, not a Intel as the other. Quite confusing to be honest. I would have hoped that Dell launched a completely new line for the opteron gear.
I would not consider this box unless I was installing at least 4 (or possibly even 6) nodes for a single deployment. If they fail, you'll have a longer down time period than most other boxes since you probably have to get it out of the rack to service it. But it's cheap and just get things done. I can imagine many high performance computing farms rubbing their hands when they see this box, it's very suitable for a Linux beowulf cluster.
You can read my review of the older SC1425.
Starting price in a usable spec: $2900 (estimated)

Dell PowerEdge 1955
The Blade server in the new family. Data processing, web servers, app servers and possibly other uses if you add the FC daughter card. Quite decent machines and as a 1855 admin I'm very pleased so far.
One thing to remember though, the 1955/1855 chassis backplane is not redunant, deploy in groups of two chassis and at least 8 servers to make them cost effective. Price excluding chassis, which usually comes in at about $3000 with redundant PSU's, DRAC, IP-KVM and a pair of switch modules.
Starting price in usable spec: $2516

Regarding the price, the starting list price on Dells website is unusable. One disk drive included, way to little RAM, no DRAC card etc. All machines have been speced out to a suitable minimal config for actuall deployment use with redundant PSU's, a PERC card, at least 2Gb RAM etc and Bronze NBD support. The size and usage of the box has also been considered, as an example the 2900 was fitted with six drives since it's main purpose is disk and I/O expandability.

Update:
For Solaris 10 support see this post.

NFS locking issues

Been quite busy at work, decommissioning some old servers.
We are trying to get rid of everything pre RHEL/CentOS 3.x. After buying a new shiny file server for our home directories (a Dell 2850 with lots of cheap disk space) I ran in to some very weird problems. Everything worked fine at first, I myself (as a XFCE user) didn’t have any problems at all, but our GNOME users ran it some serious problems. When logging in to GNOME the desktop did not start properly, gnome-panel looked like a grey bar without any tools and the desktop menu didn’t appear on right click. Simply, nothing in GNOME worked. Not in Centos, not in Fedora, not even JDS in Solaris worked.
After installing the GNOME debuginfo packages for Linux it appeared to be some weird NFS locking problem (haven’t we all hear that one before) in gconfd. NLM auth lock calls got denied. Hard to troubleshoot why and how at the time, we simply remounted the NFS shared with the nolock mount option in Linux and the sort of undocumented mount option llock in Solaris.
NFSv3 is just a mostly irritating patch work of addons, statd, lockd etc etc.
Hate NFSv3, why didn’t I include a NFSv4 migration while we where at it.
The nolock "hack" will have to do for now, I’ll will try to troubleshoot the problem later on.

fstab line in Linux:

fs1:/export/home  /export/home nfs rsize=16384,wsize=16384,intr,nolock

I'll try to post some rather interesting Oracle 10gR2 Sun Fire T200 server benchmarks later on this week. I've found a rather good benchmarking tool called Swingbench. Written in Java and easy to use. It comes with a data loading utility to populate the database with nice bogus data and then has a few tools to simulate different workloads.
Been reading a lot about file system tuning recently as well, ext3 tuning parameters and Oracles filesystemio_options parameter, need to dig up a good test box for that so I can evaluate the options. My current two disk machine doesn't cut it for I/O load testing.

Finally a quick movie recommendation.
Bought a new LCD TV the other day so I've been overindulging in movies recently.


And a big thanks to Lukas Smith for sending me a few DVDs from my wish list.