Write that page helps me to remind the things.

1. On alert.log there was nothing
2. The file indicated by the message in application log (#201) did not exist on the database

After a while we was able to reproduce the problem, it was a query with a group by on a large data set. After a couple of test my intuition was that the problem were the TEMPORARY tablespace, so i’ve created a new TEMPORARY tablespace, i’ve setted it as new default temporary tablespace ad re-tried the test with success. It is clear that there is a bug that cause Oracle db (9.2.0.1) to crash with particular corruption on TEMPORARY tablespace.

On Alert.log we found:

KCF: write/open error block=0x4351 online=1
file=2 O:\ORACLE\ORADATA\GEOP\UNDOTBS01.DBF
error=27070 txt: 'OSD-04016: Error queuing an asynchronous I/O request.
O/S-Error: (OS 2) The system cannot find the file specified.'
Automatic datafile offline due to write error on
file 2: O:\ORACLE\ORADATA\GEOP\UNDOTBS01.DBF
Tue Jul 10 02:08:42 2007
Errors in file o:\oracle\admin\geop\udump\geop_ora_844.trc:
ORA-00376: file 2 cannot be read at this time
ORA-01110: data file 2: 'O:\ORACLE\ORADATA\GEOP\UNDOTBS01.DBF'
ORA-00372: file 2 cannot be modified at this time
ORA-01110: data file 2: 'O:\ORACLE\ORADATA\GEOP\UNDOTBS01.DBF'

where the two lines

ORA-00376: file 2 cannot be read at this time
ORA-01110: data file 2: 'O:\ORACLE\ORADATA\GEOP\UNDOTBS01.DBF'

were repeated thousands of times.

at same time in windows event viewer:

Event Type: Warning
Event Source: Ftdisk
Event Category: None
Event ID: 50
Date: 07/07/2007
Time: 07:55:15
User: N/A
Computer: GEOCALL2
Description:
{Lost Delayed-Write Data} The system was attempting to transfer file data from buffers to \Device\HarddiskVolume5. The write operation failed, and only some of the data may have been written to the file.
Data:
0000: 00 00 04 00 02 00 56 00 ......V.
0008: 00 00 00 00 32 00 04 80 ....2..€
0010: 00 00 00 00 00 00 00 00 ........
0018: 00 00 00 00 00 00 00 00 ........
0020: 00 00 00 00 00 00 00 00 ........
0028: 0e 00 00 c0 ...À


and repeated messages


Event Type: Warning
Event Source: Disk
Event Category: None
Event ID: 51
Date: 07/10/2007
Time: 02:21:01
User: N/A
Computer: GEOCALL2
Description:
An error was detected on device \Device\Harddisk3\DR3 during a paging operation.
Data:
0000: 04 00 22 00 01 00 72 00 .."...r.
0008: 00 00 00 00 33 00 04 80 ....3..€
0010: 2d 01 00 00 0e 00 00 c0 -......À
0018: 00 00 00 00 00 00 00 00 ........
0020: 00 00 00 00 00 00 00 00 ........
0028: 04 00 00 00 03 00 00 00 ........
0030: 00 00 00 00 2a 00 00 00 ....*...
0038: 00 08 00 00 00 00 00 00 ........
0040: 2a 00 02 4f db 2f 00 00 *..OÛ/..
0048: 08 00 ..


I’ve to say that this Oracle installation is not very lucky, messages of disk problems in event viewer sometimes returns, but hardware vendor tell us that there are no problems on the hardware.

Another thing that i’ve to remember is to read with my eyes the alert.log. In fact i was called by a collaborator and i did not see the line

Automatic datafile offline due to write error on


Immediately i’ve thought a corruption on the file, i’ve created a new UNDO tablespace, i’ve changed UNTOTBS parameter to point to the new tablespace. Then we tried to remove the old tablespace but we got a message that a rollback segment was “active”. In V$TRANSACTION there was no records. I was not able to understand why Oracle was telling us that. The database could be opened but the application on one step was still given an error message by Oracle stating that old undo tablespace datafile were not available. So we decided to restore from backup the tablespace, we recovered. After that i onlined the tablespace, i made a “select count(*)” from a table (the table used by application that had given the error). After that i’ve been able to drop tablespace with datafile.

Conclusion

My description has been confused but the conclusion, and the lesson i’ve learned is that UNDO TABLESPACE may contain data needed to the integrity of the database. I think that is the case of “delayed block cleanout”. If there are active trasactions that is obvious and it is visible by V$TRANSACTION system view, but in the case of “delayed block cleanout” i think that information is not easily available.

So, i’m back to what Tom has already said: there is no a better os, you must choose that with wich YOU are more comfortable, there are no other scientific and absolute motivations to prefer one over another. I feel better with Linux, but i’ve learned a lot of things about Windows that make me better.

Personally i’ve to say that Windows has good performance, but Oracle threaded architecture on Windows it seems not so good as multi-process architecture on Unix.

Pro-Unix i want remember always a more consolidated architecture. (Unix is here from 1970 )

I completely agree with the quote “Complexity is the enemy of availability” , i’m conservative. So i think that technologies pushed to simplify management may became a boomerang.  On the other hand i think that RAC, with 10g standard edition has reason to be. It give us a little scalability at a competitive cost.

What mogens says in a comment is the real point:

“For political reasons you might have to implement all sorts of things, and I still haven’t found an effective way of preventing that from happening.”

Marketing and politics really drive our (mine) managers so i have to implement things that for me give no advantages.

however i’ve to say that my little experience with RAC, not about HA, is at last good.  What remains are all the bugs that we can encounter, with RAC as with Standalone instance.

Let’s start from the beginning, we have an application developed and tuned on Oracle 9iR2 with RBO. We started with 9.2.0.1 about four years ago and there were not great problems with performance. Application is OLTP. RBO has a great advantage: stabilitity. On the other hand tuning an application with CBO is complex. One day our manager decided to make an installation of a two node RAC standard edition, with 10g (applications will have had 400 users). It was a 10gR1 we made the great jump in to the dark migrating to CBO. Things gone well (apart from installation). Then we made another jump in the dark with installation of 10gR2 and there started the problems. With my experience it seems that optimizer in 10gR2 is worse than optimizer in 10gR1.

The problem

Well, i’m not guilty, our application has a table (i will call it X) that can join alternatively with other three tables (i repeat, i’m not guilty, i think that not a good idea and a bad design, i will call such tables A, B and C). The table X as three columns ID_A that is foreign key (?) with table A, ID_B that is foreign key with table B and ID_C that is foreign key with table C. Our application is a product, customizable, so for a client happens that this table has milions of records that join only table C, foreign keys for tables A and B have value “-1″. So application make a query like this:

SELECT * FROM X
WHERE
ID_A = n1 or ID_B = n2 or ID_C = n3

Where n1>0, n2>0, n3>0 and in table X all values for ID_A and ID_B are “-1″. There are three indexes on columns ID_A, ID_B and ID_C. Using histograms optimizer knows that the thee values n1,n2,n3 (>0) are very selective so it uses the three indexes on 10gR1. Also on 10gR2 if we use litteral values. Our application is a java application that uses Bind Variables. So it happened that suddenly in 10gR2 optimizer decided to do a triple full scan on the table making database server hang on I/O (table has over 20 milion of records). I think that it is a problem with bind peeking, but i’m not been able to reproduce it, so we tried STORED OUTLINES to stabilize execution plan using indexes.

So i encountered the BUG 4323868, optimizer as required from hints of stored outlines used indexes but making index full scan instead of index range scan, making database server hang on I/O.

It is interesting to note what Oracle has made, i’ve not found patch for this bug, with patchset 10.2.0.3 Oracle has introduced “code improvement” adding two new hints: INDEX_RS_ASC and INDEX_RS_DESC