梦想有多远

redhat 5.5 multipath

bosonmaster — Fri, 13 Apr 2012 16:41:20 +0800

REDHAT 5以后不能再使用 qla的多路径软件，要使用系统自带的multipath工具，在系统启动的时候，会报如下错，这个和本地硬盘划分的分区个数有关（错误条数）
Apr 13 15:24:49 tgdb1 kernel: device-mapper: table: 253:0: multipath: error getting device
Apr 13 15:24:49 tgdb1 kernel: device-mapper: ioctl: error adding target to table
Apr 13 15:24:49 tgdb1 kernel: device-mapper: table: 253:0: multipath: error getting device
Apr 13 15:24:49 tgdb1 kernel: device-mapper: ioctl: error adding target to table
Apr 13 15:24:49 tgdb1 kernel: device-mapper: table: 253:115: multipath: error getting device
Apr 13 15:24:49 tgdb1 kernel: device-mapper: ioctl: error adding target to table
Apr 13 15:24:49 tgdb1 kernel: device-mapper: table: 253:115: multipath: error getting device
Apr 13 15:24:49 tgdb1 kernel: device-mapper: ioctl: error adding target to table

问题很简单，在mulutipath启动的时候检测到本地硬盘只有一条路径，所以报错，所以只要在multipath.conf里排出本地硬盘和分区就可以了。
blacklist{devnode “^sda”
devnode “^sda[1-3]}
可以看下官方的解释：http://sources.redhat.com/lvm2/wiki/MultipathUsageGuide，第9条，以前报的时候，知道没问题，懒的搞，今天处理了下。
When multipath/multipathd starts you might see a message(s) like

device-mapper: table: 253:0: multipath: error getting device
device-mapper: ioctl: error adding target to table
in console or /var/log/messages. This is due to dm-multipath trying to create multipath devices for your root device and/or other devices that are already mounted or opened.
You can avoid this by adding a blacklist stanza in your /etc/multipath.conf file for those devices that generate these errors.

ASM的limits(留看)

bosonmaster — Thu, 23 Feb 2012 08:52:31 +0800

ASM imposes the following limits:

63 disk groups in a storage system

10,000 ASM disks in a storage system

2 terabyte maximum storage for each ASM disk (the Bug 6453944 allowed larger sizes, but that led to problems, see Note 736891.1 “ORA-15196 WITH ASM DISKS LARGER THAN 2TB”)

40 exabyte maximum storage for each storage system

1 million files for each disk group

2.4 terabyte maximum storage for each file

redhat 5.5网卡绑定驱动问题

bosonmaster — Sat, 04 Feb 2012 11:45:48 +0800

最近在做一套ORACLE RAC 系统，在做网卡绑定的时候，发现启动以后，网络不通，手动DOWN UP一下就OK，我的LINUX系统版本是REDHAT AS 5.5,经确认为网卡驱动问题，更新驱动后，启动正常，硬件为HP DL580G7，以下是驱动连接，具体根据自己网卡型号下载
DL580G7集成的NC375i网卡的驱动

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=4142916&swItem=MTX-95d32167239142288ce7da6c7e&prodNameId=4142792&swEnvOID=4004&swLang=8&taskId=135&mode=4&idx=2

通过您的描述，您的机器还添加了一块光纤接口的千兆网卡，在DL580G7的选件列表内，查到有一款NC373F的网卡与您的描述符合，下载链接如下。因您未能提供网卡型号，所以建议您确认添加的确实是NC373F后再进行安装。

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=329290&prodSeriesId=1155247&swItem=MTX-5f19b69217e84228a9c981a8ba&prodNameId=1155223&swEnvOID=4004&swLang=8&taskId=135&mode=4&idx=1

安装驱动过程如下：
1 安装下载的SRC.RPM包 rpm -ivh
2 build 二进制RPM包 rpmbuild -bb /usr/src/redhat/SPECS/*.spec 具体什么名字自己确认
3 确认系统时候有其他版本的包 rpm -q kmod 有的话 rpm -e 卸载
4下载已经加载的模块，modprobe -r netxen_nic
5 安装RPM包 rpm -ivh /usr/src/redhat/RPMS/*.rpm 有依赖报错加 --nodeaps
6 确认模块版本是否为更新的版本后加载modinfo nx_nic modprobe nx_nic

X11TransSocketINETConnect() ：can’t get address for localhost

bosonmaster — Mon, 28 Nov 2011 09:24:48 +0800

用XMANAGER XSTART 连接LINUX服务器启用图形界面时，报错X11TransSocketINETConnect() ：can’t get address for localhost，最终确认为
/etc/hosts文件缺少127.0.0.1地址，添加上就OK

喜迎2011年

bosonmaster — Wed, 02 Feb 2011 23:57:26 +0800

祝大家2011年，身体健康，合家欢乐!

BOSON给各位拜年了，happy new year!

ora-39095

bosonmaster — Mon, 06 Sep 2010 12:48:51 +0800

今天在导出数据文件的时候，倒着倒着就报错了
ORA-39095:Dump file space has been exhausted: Unable to allocate string bytes
Cause: The Export job ran out of dump file space before the job was completed.
Action: Reattach to the job and add additional dump files to the job restarting the job.

从报错的字面意思，一开始以为系统空间问题，但是空闲的空间还是很多的，根据提示增加了个dumpfile但是还是报错，后来查了下METALINK，可能是我的PARALLEL太高了。我从8改为4就好了。或者也可以通过增加DUMPFILE，也能解决问题。 dumpfile=hisdeliverxU%就OK。记录下

4个监听的问题

bosonmaster — Thu, 12 Aug 2010 09:24:08 +0800

今天启动一个测试库，发现连接不上客户端，一看4个监听，就想到之前看到过文章，说10.2.0.1上有这个问题，listener.ora加入以下参数解决
SUBSCRIBE_FOR_NODE_DOWN_EVENT_ =OFF 随便记录下，以备查找

impdp and optimizer_mode rule

bosonmaster — Fri, 12 Mar 2010 23:28:55 +0800

因为和应用有关，公司的工具一直用imp exp ，最近在客户那是用impdp导入数据的时候，无法导入，报如下错：
Impdp fails with the following errors:
UDI-00008: operation generated ORACLE error 31626
ORA-31626: job does not exist
ORA-06512: at "SYS.KUPC$QUE_INT", line 536
ORA-25254: time-out in LISTEN while waiting for a message

一开始怀疑，主表的问题，但是删除重新导入还是不行，最后确认是和数据库的优化模式有关，因为我们的应用，数据库的优化模式为RULE，改为除RULE其他模式后，导入正常，文档： 577562.1有描述：
In a RAC database a full Data Pump export fails with:

ORA-39097: Data Pump job encountered unexpected error -1422
ORA-39065: unexpected master process exception in DISPATCH
ORA-01422: exact fetch returns more than requested number of rows
这个类似的错误也是这个原因引起的

linux ipcs max sharememory

bosonmaster — Tue, 09 Mar 2010 14:29:38 +0800

最近同事遇到一个LINUX上共享内存段的问题，一般我们会根据系统内存的大小来定义系统支持的最大内存段大小，kernel.shmmax ，检查了这些设置都没问题，系统版本也没啥问题，最后经老熊提醒确认为NUMA的问题
ipcs -m

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00000000 98304 gdm 600 393216 2 dest
0x00000000 1310721 oracle 640 1543503872 31
0x00000000 1343490 oracle 640 2835349504 31
0x00000000 1376259 oracle 640 2835349504 31
0x00000000 1409028 oracle 640 2852126720 31
0x00000000 1441797 oracle 640 2835349504 31
0x1714b88c 1474566 oracle 640 2097152 31
设置了如下参数后就解决了次问题
alter system set "_enable_numa_optimization"=false scope=spfile;
有关NUMA的问题，可以看文档：759565.1

linux oracle rac oprocd reboot

bosonmaster — Sun, 07 Feb 2010 19:14:03 +0800

最近在客户升级10.2.0.4外加，CRS PSU2 DB PSU3，升级完后，系统不定时发生重启，发生重启期间都有大文件操作，根据OSW记录的系统状态情况，当时内存剩余比较少，但是又有某些重启时刻系统重启时，内存也有不少剩余，每次发生系统重启的时候，系统收到的信息都是SysRq:reseting ，然后系统就重启了，中间测试了很多种情况，包括写在PSU2 压力测试都是如此
10.2.0.4比10.2.0.3在监控节点间的状态的时候，特别LINUX平台，多了一个oprocd进程，用于检测节点状态，如果发生如下情况，就会重启系统
A problem detected by the OPROCD process. This can be caused by 4 things:

1) An OS scheduler problem.
2) The OS is getting locked up in a driver or hardware.
3) Excessive amounts of load on the machine, thus preventing the scheduler from
behaving reasonably.
4) An Oracle bug.

并且如果是OPROCD进程导致系统重启，那么会看到SysRq:reseting信息，在如下位置有oprocd日志
/etc/oracle/oprocd or /var/opt/oracle/oprocd
oprocd默认启动oprocd run -t 1000 -m 500 单位为毫秒，默认允许延迟时间为1S，如果1S内没有响应，那么0.5秒后重启系统，也就说OPROCD检测问题后，允许的延迟时间是1S，1S后延迟后，0.5秒后OPROCD进程就会掉用脚本重启系统，我们这次遇到的都是SysRq:reseting，所以是OPROCD进程重启的系统，我们测试了很多次。
10.2.0.4以前LINUX 上是通过hangcheck timer模块来检测的，默认的延时
   * 9i: Assuming the default setting of "oracm misscount" is set to 220 seconds:
   hangcheck_tick=30 hangcheck_margin=180 hangcheck_reboot=1
   * 10g/11g: Assuming the default setting of "CSS misscount" is set to either 30 or 60 seconds:
   hangcheck_tick=1 hangcheck_margin=10 hangcheck_reboot=1
还有以下几个进程可能重启系统

1 ocssd进程

ocssd进程引起节点重启的时候，会有类似如下日志：
Rebooted for Cluster Integrity 在系统日志里，比如linux /var/log/message hp-ux syslog,CRS日志里有如下类似记录
Sun: /var/adm/messages
HP-UX: /var/adm/syslog/syslog.log
Tru64: /var/adm/messages
Linux: /var/log/messages
IBM: /bin/errpt -a > messages.out

Network failure or latency between nodes. It would take at least 30 consecutive
missed checkins to cause a reboot, where heartbeats are issued once per second.

Example of missed checkins in the CSS log:

WARNING: clssnmPollingThread: node <node> (1) at 50% heartbeat fatal, eviction in 29.100 seconds
WARNING: clssnmPollingThread: node <node> (1) at 75% heartbeat fatal, eviction in 14.960 seconds
WARNING: clssnmPollingThread: node <node> (1) at 75% heartbeat fatal, eviction in 13.950 seconds

The first thing to do is find out if the missed checkins ARE the problem or are a
result of the node going down due to other reasons. Check the messages file to see
what exact time the node went down and compare it to the time of the missed checkins.

- If the messages file reboot time < missed checkin time then the node eviction was
likely not due to these missed checkins.

- If the messages file reboot time > missed checkin time then the node eviction was
likely a result of the missed checkins.

- Problems writing to or reading from the CSS voting disk.

Example of a voting disk problem in the CSS log:

ERROR: clssnmDiskPingMonitorThread: voting device access hanging (160008 miliseconds)

- Lack of CPU resources. There are some situations which will appear to be missed
heartbeat issues, however turn out to be caused by a user running a high
sustained load average. When a machine gets too heavily loaded, the scheduling
reliability can be bad. This could cause CSS to not get scheduled in time and
thus CSS cannot get its work done. If this happens, the node is declared
not-viable for cluster work and is evicted.

- A problem with the executables (for example, removing CRS Home files)

- Misconfiguration of CRS. Possible misconfigurations:

- Wrong network selected as the private network for CRS (confirm with CSS log,
/etc/hosts, and ifconfig output). Make sure it is not the public or VIP
address. Look in the CSS log for strings like...
clsc_listen: (*) Listening on
(ADDRESS=(PROTOCOL=tcp)(HOST=dlsun2046)(PORT=61196))

- Putting the CSS vote file on a Netapp that's shared over some kind of public
network or otherwise excessively loaded/unreliable network. If this is the
case, you are likely to see the following message in the CSS logfile:

ERROR: clssnmDiskPingThread(): Large disk IO timeout * seconds.

If you ever see this error, then it's important to investigate why the disk
subsystem is unresponsive.

See section 3.2 for information on how to correct common misconfiguration
problems.

- Killing the "init.cssd fatal" process or "ocssd" process.

- An unexpected failure of the OCSSD process, this can be caused by any of the
above issues.

- An Oracle bug. Known bugs that can cause CSS reboots:

3 第三个进程如下 oclsomon
A problem detected by the OCLSOMON process. This can be caused by 4 things:

1) A thread(s) within the CSS daemon hung.
2) An OS scheduler problem.
3) Excessive amounts of load on the machine, thus preventing the scheduler from
behaving reasonably.
4) An Oracle bug.

更加信息的资料可以看文档：265769.1 726833.1 395878.1