The system call nanosleep uses the system call cv_timedwait_sig to wait. Starting with Solaris 10 update 8 the system call cv_timedwait_sig uses the system call cv_timedwait_sig_hires to wait which then uses gethrtime() to schedule the wake up time. As this high resolution timer can jump on Solaris 10 running on top of a VMware ESX 3.5, VMware vSphere 4.x, VirtualBox virtual maschine with more than 1 CPU assigned, it can happen if the sleep exactly starts when such a jump occurs, that the end time is so far in the future, that it seems to system call is hanging for ever. As soon as one uses truss, pstack, jstack, ... to analyze the live process, the nanosleep call will wake up on a signal and continue.
To see if the high resolution timer is making jumps you can use the following small program gethrtime_test.c to test:

#include <stdio.h>
#include <stdlib.h>
#include <sys/time.h>

int main(int argc, char** argv)
{
     hrtime_t start, end;
     long iters = 1000000000;
     hrtime_t delta;
     delta=10000000;
 
     if (argc > 1) iters = atol(argv[1]);
     if (argc > 2) delta = atol(argv[2]);
     printf("%ld iterations\n", iters);
 
     start = gethrtime();
 
     long i;
     for (i = 0; i < iters; i++)
     {
         end = gethrtime();
 
         if ( start > end || start+delta < end)
             printf("%ld:\n  start %lld\n  end   %lld\n  diff  %lld\n",
                 i, start, end, (end-start));
 
         start=end;
     }
     exit(0);
}

Create the file called gethrtime_test.c and compile it

root@hostname# gcc -o gethrtime_test gethrtime_test.c

Then let it run.

root@hostname# ./gethrtime_test
1000000000 iterations

You should not see any message like

5750624:
  start 415551665195
  end   415150888326
  diff  -400776869
6387810:
  start 416494513021
  end   416509397658
  diff  14884637

Especially negative jumps should never happen.

If you see negative jumps, you can set the following paramters in a VMware virtual machines vmx file and reboot to avoid this:

VMware ESX 3.5

monitor_control.disable_tsc_offsetting=TRUE
monitor_control.disable_rdtscopt_bt=TRUE

VMware vSphere 4.x

timeTracker.forceMonotonicTTAT=TRUE

On VirtualBox I do not know any solution yet.

[ view entry ] ( 912 views )   |  permalink  |  print article  |   ( 3 / 2624 )

---

When running a Oracle Databases on a UFS or ZFS filesystem on Solaris 10 X86 on VMware virtual machine, the default values of all the stacks involved (Oracle Database, Solaris, VMware) can cause a database corruption, each time the VMware virtual machine is powered off without cleanly shut down the operating system. This happens if the power off button in virtual center is used to forcibly power off the virtual machine or if there is a basic hardware failure causing the physical node to crash.
This is true for Solaris 10 upto u10, VMware ESX 3.5 through to vSphere 4.1, UFS and ZFS, Oracle 10 through to Oracle 11.

To avoid this there are 2 solutions:

1) use the forcedirectio option for all filesystems where database files (redo log members, data files, controlfiles) in /etc/vfstab:

/dev/md/dsk/d20   /dev/md/rdsk/d20  /u01  ufs 3   yes  forcedirectio

2) use the oracle init parameter filesystemio_options to force directio on all database instances:

oracle@hostname> sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Fri Jan 13 13:32:08 2012

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Connected to an idle instance.
SQL> alter system set filesystemio_options=setall scope=spfile;

System altered.

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area  267825152 bytes
Fixed Size		    1335924 bytes
Variable Size		  130026892 bytes
Database Buffers	  130023424 bytes
Redo Buffers		    6438912 bytes
Database mounted.
Database opened.
SQL> quit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

[ view entry ] ( 3125 views )   |  permalink  |  print article  |   ( 2.9 / 4192 )

---

<<First <Back | 1 | 2 | 3 |

Doerzbach Engineering GmbH, Himmelrichstrasse 14, 6340 Baar, Switzerland - Page Generated in 0.0345 seconds | Site Views: 156209