This is a set of notes I took while trying to figure out how to get NFS to work on my RHEL/CentOS 5.3 boxes. They were to serve as NFS host for CentOS 5.4 installation images. Even though this is on RHEL/CentOS, it should apply to many Linux/Unix systems since this NFS.
The process brought me to a understanding of NFS configuration and troubleshooting better than ever. Before I go into the details, here is a list of common NFS problems on HP site. Even the details are different due to HP UX vs Solaris vs Linux, the list is a good kick off point to understand the common problem of NFS:
Now on with my jouney that represetns but one common NFS problem,
i.e. bad firewall configuration:
1. Problem: While installing CentOS 5.4, booting from the boot CD while using "linux askmethod", I can choose to install with various methods. One of the method I really want to use is NFS hosted Image. Save a few CDs & less mess. I also want to also use the kickstart script (modified, of course) I already have when I install CentOS 5.3 on that box to practice kickstart script installation. NFS would be the network installation source.
2. Starting NFS on RHEL/CentOS 5.3 boxes:
This is very easy to do by running "
system-config-services" command, or via GUI:
System -> Administration -> Server Setting -> Services
Both bring up the same "Service Configuration" GUI. Then just check "nfs" and click start.
Alternatively, we can start NFS from command line:
[root@server ~]$ chkconfig --level 45 nfs on
[root@server ~]$ service nfs start
and that would start nfs and related daemon.
Note: On systems without chkconfig utility (I think Solaris 9 would be an example, please verify), you need to start the script /etc/init.d/nfs and add the symbolic link or hard link to /etc/rc*.d directory of the run level you are looking to start nfs.
Now make sure that the iso image is in the right dir:
[root@server ~]$ ls -l /opt/inst/CentOS-5.4/
Wait. How do we tell Linux to share /inst? Add this to /etc/exports file:
/inst *(ro, sync)
OK. Export by running
[root@server ~]$ exportfs -a
Or, restart NFS:
[root@server ~]$ service nfs restart
[root@server ~]$ showmount -e
Export list for server:
/inst *
/mnt/loop *
Good, we can see the dir being exported now. Are we set now?
Nope! The installation client still cannot see the install media. So we need to figure out why it doesn't work.
3. Trouble Shooting NFS
3.1 Is NFS running? Obviously,
[root@server ~]$ ps -ef | grep -i nfs
would give some idea about the process too. It is more useful to
[user@client ~]$ /usr/sbin/rpcinfo -p server
from a client's terminal window. If you get something like this
program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 902 status
100024 1 tcp 905 status
100011 1 udp 774 rquotad
100011 2 udp 774 rquotad
100011 1 tcp 777 rquotad
100011 2 tcp 777 rquotad
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100021 1 udp 58882 nlockmgr
100021 3 udp 58882 nlockmgr
100021 4 udp 58882 nlockmgr
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100021 1 tcp 49080 nlockmgr
100021 3 tcp 49080 nlockmgr
100021 4 tcp 49080 nlockmgr
100005 1 udp 808 mountd
100005 1 tcp 811 mountd
100005 2 udp 808 mountd
100005 2 tcp 811 mountd
100005 3 udp 808 mountd
100005 3 tcp 811 mountdThan NFS and related daemons with all their previous versisons (for backward compatibility, I guess) are running on the server, and the client can see them too. If instead you get only
program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 902 status
100024 1 tcp 905 status
then you can get to RPC information but NFS is not running.
However, if you get
rpcinfo: can't contact portmapper: RPC: Remote System error - No route to host
You might say "What do you mean? I can ping my server and ssh to it?
No route to host?" That just means your server's firewall is blocking portmapper port. All NFS stuff might be running just fine, as you might have done the same command on server itself to verify. This is a tighten security and shooting yourself in the foot type of deal.
OK. Let's add 111 tcp to the allowed port (see upcoming Firewall & iptables post) for details. Bingo, rpcinfo works on client now and we can verify that nfs & friends are running.
3.2 Firewall Wows and It's about Time to learn
iptables
As a quick check, I disabled the firewall and
walla! it worked.
After installing one box, turn the firewall back on. Add all the ports to firewall exception rule (again, use GUI is quick & easy). Verify that it works. Since I have a dual boot server, I need to change over to Windows to do some work. This newly built system is going to be a dedicated CentOS server, I promise.
Next morning, boot up my server. Again, doing a showmount from a client:
[user@client ~]$ showmount -e serverIt is not work again! What is going on?
Run rpcinfo on the server, and now I see
[user@server ~]$ /usr/sbin/rpcinfo -p server
program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 902 status
100024 1 tcp 905 status
100011 1 udp 694 rquotad
100011 2 udp 694 rquotad
100011 1 tcp 697 rquotad
100011 2 tcp 697 rquotad
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100021 1 udp 41214 nlockmgr
100021 3 udp 41214 nlockmgr
100021 4 udp 41214 nlockmgr
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100021 1 tcp 32848 nlockmgr
100021 3 tcp 32848 nlockmgr
100021 4 tcp 32848 nlockmgr
100005 1 udp 708 mountd
100005 1 tcp 711 mountd
100005 2 udp 708 mountd
100005 2 tcp 711 mountd
100005 3 udp 708 mountd
100005 3 tcp 711 mountd
The port has changed. That is why. nfs keeps the same port (as they should since the reservation is done in /etc/services file, but most of the other daemons changes port number. Is there any reason not to reserve ports for these daemons? Say, 2050, 2051 for mountd, 2052 for status, etc?
Add all the ports currently active. Then I can mount these exported dir from another client, say client2. Try to build client1 again, and still cannot access. Try boot client1 from HD (currently running RH9, very old). I can do rpcinfo & showmount from the client but I got error message:
mount: RPC: Timed out
and it failed. Is there a network issue?
NFS4 uses only 2049 but NFS2, 3 still requires portmap, lockd, mountd and all those daemons to run on individual ports. It looks like RHEL/CentOS remote NFS installation uses older version of the protocol. It makes the service harder to secure and harder to use because these other services uses random ports by default.
Problem Resolved
I went back to read RHCE prep book by Michael Jang. It did mentioned that among NFS daemons, only portmapper (111) & nfs (2049) uses fixed port numbers (they are in /etc/services file). The other daemons: statd, lockd, mountd, all uses random ports. And there is nothign wrong with fixing them in /etc/services. Once that is done, the behavior of NFS becomes predictable. Once this is done. You can open up all these prots on firewall and get NFS to work.
An alternative to this is to use HTTP remote install. This requires the server to run an httpd, Apache. The procedure is similar to NFS in concepts and you only need to open up one port of your choice (set in httpd.conf). There is a little bit more work than using NFS since you cannot use iso image, but instead, do a loop mount would take care of the source.
[root@server /opt/inst/CentOS-5.4]$ mount -o loop CentOS-5.4-i386-bin-1of6.iso /mnt/inst/CentOS-5.4
That took care of getting the files into the dir.