Hacktopia » Solaris 10

How to mirror ZFS root disk on a sparc system.

admin — Wed, 24 Mar 2010 18:43:23 +0000

First we look to see that everything looks good on our current drive that we are booted off of. I would also like to note that there is no UFS on this system at all. We are booting entirely off of the ZFS pool rpool.

# zpool status
pool: rpool
state: ONLINE
scrub: none requested
config:

NAME        STATE     READ WRITE CKSUM
rpool       ONLINE       0     0     0
c0t0d0s0 ONLINE       0     0     0

errors: No known data errors

Now verify that the two disks that you want to mirror looks the same at the slave level.

# format
Searching for disks…done

AVAILABLE DISK SELECTIONS:
0. c0t0d0
/pci@1f,0/ide@d/dad@0,0
1. c0t2d0
/pci@1f,0/ide@d/dad@2,0
Specify disk (enter its number): 0
selecting c0t0d0
[disk formatted, no defect list found]
/dev/dsk/c0t0d0s0 is part of active ZFS pool rpool. Please see zpool(1M).
/dev/dsk/c0t0d0s2 is part of active ZFS pool rpool. Please see zpool(1M).

FORMAT MENU:
disk       – select a disk
type       – select (define) a disk type
partition – select (define) a partition table
current    – describe the current disk
format     – format and analyze the disk
repair     – repair a defective sector
show       – translate a disk address
label      – write label to the disk
analyze    – surface analysis
defect     – defect list management
backup     – search for backup labels
verify     – read and display labels
save       – save new disk/partition definitions
volname    – set 8-character volume name
!     – execute , then return
quit
format> ver

Primary label contents:

Volume name = <        >
ascii name =
pcyl        = 38792
ncyl        = 38790
acyl        =    2
nhead       =   16
nsect       =   63
Part      Tag    Flag     Cylinders         Size            Blocks
0       root    wm       0 – 38789       18.64GB    (38790/0/0) 39100320
1 unassigned    wm       0                0         (0/0/0)            0
2     backup    wm       0 – 38789       18.64GB    (38790/0/0) 39100320
3 unassigned    wm       0                0         (0/0/0)            0
4 unassigned    wm       0                0         (0/0/0)            0
5 unassigned    wm       0                0         (0/0/0)            0
6 unassigned    wm       0                0         (0/0/0)            0
7 unassigned    wm       0                0         (0/0/0)            0

format> disk 1
selecting c0t2d0
[disk formatted, no defect list found]
format>

Notice that both disks have 0 – 38789 Cylinders. So the disks look good, now we can attach the second disk.

# zpool attach rpool c0t0d0s0 c0t2d0s0
#
# zpool status
pool: rpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress for 0h0m, 3.28% done, 0h24m to go
config:

NAME          STATE     READ WRITE CKSUM
rpool         ONLINE       0     0     0
mirror      ONLINE       0     0     0
c0t0d0s0 ONLINE       0     0     0
c0t2d0s0 ONLINE       0     0     0

errors: No known data errors

WoW that was easy to mirror two disks with ZFS. No need to set up data stores or anything like that. The fact that the zpool status output specify “mirror” means the disks are mirrored and not striped. If you have multiple zfs pools you can specify zpool status POOLNAME.

Now to try to distroy the disk, ha, ha, ha!

# dd if=/dev/random of=/dev/rdsk/c0t2d0s0
969+0 records in
969+0 records out

Now if that disk was on a normal RAID utility stuch as solstice disk suite SDS or even hardware raid then both disks would be destroyed because they operate at the block level, and will copy the random blocks back do c0t0d0s0

# zpool status
pool: rpool
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using ‘zpool online’.
see: http://www.sun.com/msg/ZFS-8000-2Q
scrub: resilver in progress for 0h1m, 10.52% done, 0h12m to go
config:

NAME          STATE     READ WRITE CKSUM
rpool         DEGRADED     0     0     0
mirror      DEGRADED     0     0     0
c0t0d0s0 ONLINE       0     0     0
c0t2d0s0 UNAVAIL      0     0     0 cannot open

errors: No known data errors

Oha happy ZFS is already working on fixing or resilvering the disk that was written to with random data.

# zpool status
pool: rpool
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using ‘zpool online’.
see: http://www.sun.com/msg/ZFS-8000-2Q
scrub: resilver completed after 0h0m with 0 errors on Tue Apr 14 12:55:48 2009
config:

NAME          STATE     READ WRITE CKSUM
rpool         DEGRADED     0     0     0
mirror      DEGRADED     0     0     0
c0t0d0s0 ONLINE       0     0     0
c0t2d0s0 UNAVAIL      0     0     0 cannot open

errors: No known data errors

Now ZFS is angry because we wrote random data to the disk.

# zpool online rpool c0t2d0s0

# installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c0t2d0s0

Enable snmp on Solaris 10

admin — Fri, 06 Nov 2009 17:44:27 +0000

So I had some trouble gettin snmp working. I edited /etc/snmp/conf/snmpd.conf with the proper trap and community for my enviroment. Every time I tried to start snmp I would get erros.

checking /var/adm/messages

Nov 6 10:09:29 pressman2 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] can’t read the directory snmpd.conf [errno: Not a directory(20)]

me trying to start snmp

bash-3.00# svcadm enable svc:/application/management/snmpdx:default

bash-3.00# svcs|grep snmp
offline 10:03:20 svc:/application/management/snmpdx:default

On the second line you can see that I check and it was not started or offline. So then it was time to consult with svcs -x. That stated that I was missing a prerequisite process that snmp relied on.

bash-3.00# svcs -x
svc:/application/management/seaport:default (net-snmp SNMP daemon)
State: disabled since Wed Oct 28 16:25:27 2009
Reason: Disabled by an administrator.
See: http://sun.com/msg/SMF-8000-05
See: snmpd(1M)
Impact: 1 dependent service is not running. (Use -v for list.)

bash-3.00# svcs |grep snmp
online 11:25:54 svc:/application/management/snmpdx:default

Now everything works like it should.

ssh: Connection closed by remote host

admin — Tue, 07 Jul 2009 15:42:07 +0000

I was trying to ssh from one Sun Soalris server to another.

# ssh remoteserver
ssh_exchange_identification: Connection closed by remote host
# ssh -v remoteserver
Sun_SSH_1.1.1, SSH protocols 1.5/2.0, OpenSSL 0x0090704f
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Rhosts Authentication disabled, originating port will not be trusted.
debug1: ssh_connect: needpriv 0
debug1: Connecting to mimi [10.20.55.222] port 22.
debug1: Connection established.
debug1: identity file /.ssh/identity type -1
debug1: identity file /.ssh/id_rsa type -1
debug1: identity file /.ssh/id_dsa type -1
ssh_exchange_identification: Connection closed by remote host
debug1: Calling cleanup 0x34ae8(0×0)
#

Err it would not let me connect. As you can see I first tried to ssh, then ssh in verbose mode. As you can see the output did not really give anything that useful.

I logged into the remote server and all it would tell me is

Jul 7 11:37:53 remoteserver sshd[21761]: refused connect from clientserver

I wold have to be connected to the serial console to see this message, I was not able to see the message anywhere else. Not in /var/adm/messages or /var/log/syslog

So what I ended up doing was adding this line to /etc/hosts.allow

p_ctminetd,sshd,bpcd,vnetd,vopied,bpjava-msvc: 10.20.55. : allow

That line allowed any host in the 10.20.55.* subnet to connect to the server now, and ssh was allowed to connect.

Enable snmp on solaris 10

admin — Tue, 26 May 2009 16:12:23 +0000

In order to start up snmp on solaris then you need to configure

/etc/snmp/conf/snmpd.conf

and start it up using smf

svcadm enable /application/management/sma

IPMP Link-based Only Failure Detection with Solaris 10

admin — Sat, 14 Feb 2009 14:51:45 +0000

I have done a fair bit of research on this topic. While most sites will show you how to set up fail over on solris 10 with multiple ip, and probe-based failure detection. Those two things can cause problems. First off, most often ip space is at a premium and you want to be able to use as few a ip as possible. So using one ip instead of three is highly appreciated. Second having to do probe-based failure detection aka pinging the router or another host on the subnet can cause problems if you have many hosts in a data center set up in a similar fashion. While mpathd can ping a host besides the router, the default behavior is for mpathd to contently be pinging the router making sure it is still there, and if it ceases to be there any more then it will fail over. This can be as often as once a second. Also multiple subnets can be served by the same router multiplying the affects. Router’s cpu are not very fast, while they might be fast at routing, this is because the routing is accomplished through the hardware, which is carried out through a diffrent CPU.

Just to state the obvious, this is best designed for up time of service, not for aggregate speed because it is a master slave set up not master master set up. You want to make sure that your two interfaces are in different cards if at all possible and connected to two separate network switches.

Lets start with the config file.

# cat /etc/default/mpathd
#
#pragma ident “@(#)mpathd.dfl 1.2 00/07/17 SMI”
#
# Time taken by mpathd to detect a NIC failure in ms. The minimum time
# that can be specified is 100 ms.
#
#FAILURE_DETECTION_TIME=2500
FAILURE_DETECTION_TIME=10000
#
# Failback is enabled by default. To disable failback turn off this option
#
FAILBACK=yes
#
# By default only interfaces configured as part of multipathing groups
# are tracked. Turn off this option to track all network interfaces
# on the system
#
TRACK_INTERFACES_ONLY_WITH_GROUPS=yes

It is pretty self explanatory with the comments that are present. It is really nice because the only other changes that have to be made are to the /etc/hostname.ce* files.

bash-3.00# cat /etc/hostname.ce0
10.36.133.113 netmask + broadcast + group mainint up
bash-3.00# cat /etc/hostname.ce4
group mainint up

if will look like the fowling

# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ce0: flags=1000843 mtu 1500 index 8
inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255
groupname mainint
ether 0:14:4f:d6:f7:b8
ce4: flags=1000843 mtu 1500 index 9
inet 10.36.133.113 netmask ffffff00 broadcast 10.36.133.255
groupname mainint
ether 0:14:4f:4a:d5:a3

You can also fail over the interface by hand by issusing the /usr/sbin/if_mpadm command.

bash-3.00# /usr/sbin/if_mpadm -d ce0
Feb 13 14:47:31 server in.mpathd[185]: Successfully failed over from NIC ce0 to NIC ce4

bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ce0: flags=89000842 mtu 0 index 8
inet 0.0.0.0 netmask 0
groupname mainint
ether 0:14:4f:d6:f7:b8
ce4: flags=1000843 mtu 1500 index 9
inet 10.36.133.113 netmask ffffff00 broadcast 10.36.133.255
groupname mainint
ether 0:14:4f:4a:d5:a3
ce4:1: flags=1000843 mtu 1500 index 9
inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255

If you ever want to make any changes, like switch which interface is primary, then it could be helpful to Wittie a quick script to help you out. Don’t forget to make it executable by issuing the command chmod u+x script

#!/usr/bin/bash
ifconfig=/usr/sbin/ifconfig

$ifconfig ce0 unplumb
$ifconfig ce4 unplumb

Make sure that you have unique unique MAC address on the server. If you do not have unique MAC address it can confuse the switch.

to check that a unique mac is set you can “ifconfig -a” and look at the MAC addres or

bash-3.00# eeprom |grep mac
local-mac-address?=true

If it comes back false then you can fix it by issusing the fowling command

eeprom "local-mac-address?"=true

Some useful inks with further reading.

You might have to be logged into sun solve for the first link to work.
http://sunsolve.sun.com/search/document.do?assetkey=1-61-228885-1
http://docs.sun.com/app/docs/doc/816-0211/6m6nc66s8?a=view