ORA-01013: user requested cancel of current operation

1) ORA-01013: user requested cancel of current operation
2) OALL8 is in an inconsistent state
3) <BEA-000627> <Reached maximum capacity of pool
####<Apr 28, 2011 3:11:07 PM CST> <Error> <JTA> <ydcrh07> <appserver03> <itsPoolJobProcessor3-0> <wlsadmin> <BEA1-2FE6FD235C9742F5B4A1> <BEA-110412> <Xid=BEA1-2FE6FD235C9742F5B4A1(11471043),Status=Committed,HeuristicErrorCode=XA_HEURHAZ,numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds since begin=17,seconds left=120,activeThread=Thread[itsPoolJobProcessor3-0,5,main],XAServerResourceInfo[weblogic.jdbc.wrapper.JTSXAResourceImpl]=(ServerResourceInfo[weblogic.jdbc.wrapper.JTSXAResourceImpl]=(state=committed,assigned=appserver03),xar=weblogic.jdbc.wrapper.JTSXAResourceImpl@168584e,re-Registered = false),SCInfo[ngenprod+appserver03]=(state=committed),properties=({weblogic.jdbc=t3://172.18.133.104:13001, START_AND_END_THREAD_EQUAL=true}),OwnerTransactionManager=ServerTM[ServerCoordinatorDescriptor=(CoordinatorURL=appserver03+172.18.133.104:13001+ngenprod+t3+, XAResources={weblogic.jdbc.wrapper.JTSXAResourceImpl},NonXAResources={})],CoordinatorURL=appserver03+172.18.133.104:13001+ngenprod+t3+) completed heuristically: (weblogic.jdbc.wrapper.JTSXAResourceImpl, HeuristicHazard, (javax.transaction.xa.XAException: ORA-01013: user requested cancel of current operation
)) >
####<Apr 28, 2011 3:12:59 PM CST> <Error> <JTA> <ydcrh07> <appserver03> <itsPoolJobProcessor3-0> <wlsadmin> <BEA1-3C23FD235C9742F5B4A1> <BEA-110412> <Xid=BEA1-3C23FD235C9742F5B4A1(19094445),Status=Committed,HeuristicErrorCode=XA_HEURHAZ,numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds since begin=20,seconds left=120,activeThread=Thread[itsPoolJobProcessor3-0,5,main],XAServerResourceInfo[weblogic.jdbc.wrapper.JTSXAResourceImpl]=(ServerResourceInfo[weblogic.jdbc.wrapper.JTSXAResourceImpl]=(state=committed,assigned=appserver03),xar=weblogic.jdbc.wrapper.JTSXAResourceImpl@57e05e,re-Registered = false),SCInfo[ngenprod+appserver03]=(state=committed),properties=({weblogic.jdbc=t3://172.18.133.104:13001, START_AND_END_THREAD_EQUAL=true}),OwnerTransactionManager=ServerTM[ServerCoordinatorDescriptor=(CoordinatorURL=appserver03+172.18.133.104:13001+ngenprod+t3+, XAResources={weblogic.jdbc.wrapper.JTSXAResourceImpl},NonXAResources={})],CoordinatorURL=appserver03+172.18.133.104:13001+ngenprod+t3+) completed heuristically: (weblogic.jdbc.wrapper.JTSXAResourceImpl, HeuristicHazard, (javax.transaction.xa.XAException: ORA-01013: user requested cancel of current operation
)) >
####<Apr 28, 2011 3:24:55 PM CST> <Error> <JTA> <ydcrh07> <appserver01> <ExecuteThread: '14' for queue: 'default'> <<anonymous>> <BEA1-1FE0A5A13D3EACFBD58D> <BEA-110412> <Xid=BEA1-1FE0A5A13D3EACFBD58D(8655262),Status=Rolled back. [Reason=javax.ejb.EJBException: nested exception is: java.sql.SQLException: No more data to read from socket],HeuristicErrorCode=XA_HEURHAZ,numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds since begin=2,seconds left=60,activeThread=Thread[ExecuteThread: '14' for queue: 'default',5,Thread Group for Queue: 'default'],XAServerResourceInfo[weblogic.jdbc.wrapper.JTSXAResourceImpl]=(ServerResourceInfo[weblogic.jdbc.wrapper.JTSXAResourceImpl]=(state=rolledback,assigned=appserver01),xar=weblogic.jdbc.wrapper.JTSXAResourceImpl@13cba33,re-Registered = false),SCInfo[ngenprod+appserver01]=(state=rolledback),properties=({weblogic.jdbc=t3://172.18.133.104:11001, START_AND_END_THREAD_EQUAL=false}),local properties=({modifiedListeners=[weblogic.ejb20.internal.TxManager$TxListener@645881, weblogic.ejb20.internal.TxManager$TxListener@1a93af6, weblogic.ejb20.internal.TxManager$TxListener@479977, weblogic.ejb20.internal.TxManager$TxListener@9b3562, weblogic.ejb20.internal.TxManager$TxListener@1c7380e, weblogic.ejb20.internal.TxManager$TxListener@15165b5, weblogic.ejb20.internal.TxManager$TxListener@b5a11c, weblogic.ejb20.internal.TxManager$TxListener@f7138e, weblogic.ejb20.internal.TxManager$TxListener@9acd51, weblogic.ejb20.internal.TxManager$TxListener@c023c9, weblogic.ejb20.internal.TxManager$TxListener@1233206, weblogic.ejb20.internal.TxManager$TxListener@1eeb6cc, weblogic.ejb20.internal.TxManager$TxListener@1bb3bee, weblogic.ejb20.internal.TxManager$TxListener@b0d328, weblogic.ejb20.internal.TxManager$TxListener@946202, weblogic.ejb20.internal.TxManager$TxListener@1b18e12, weblogic.ejb20.internal.TxManager$TxListener@190d056]}),OwnerTransactionManager=ServerTM[ServerCoordinatorDescriptor=(CoordinatorURL=appserver01+172.18.133.104:11001+ngenprod+t3+, XAResources={weblogic.jdbc.wrapper.JTSXAResourceImpl},NonXAResources={})],CoordinatorURL=appserver01+172.18.133.104:11001+ngenprod+t3+) completed heuristically: (weblogic.jdbc.wrapper.JTSXAResourceImpl, HeuristicHazard, (javax.transaction.xa.XAException: No more data to read from socket)) >
####<Apr 28, 2011 3:24:55 PM CST> <Error> <JDBC> <ydcrh07> <appserver01> <ExecuteThread: '14' for queue: 'default'> <<anonymous>> <BEA1-203FA5A13D3EACFBD58D> <BEA-001112> <Test "select count(*) from dual" set up for pool "JDBC-Connection-Pool-for-nGen-Application" failed with exception: "java.sql.SQLException: OALL8 is in an inconsistent state".>
1) Some somments

From the appserver01's myserver.log, there was warning before the above ora error:

####<Apr 26, 2011 4:15:45 PM CST> <Info> <Common> <ydcrh07> <appserver03> <itsPo
olJobProcessor1-0> <wlsadmin> <BEA1-2187FAE3EC9842F5B4A1> <BEA-000627> <Reached
maximum capacity of pool "JDBC-Connection-Pool-for-nGen-Application", making "0"
new resource instances instead of "2".>

which indicate the itsPoolJobProcessor1-0 cannot obtain jdbc connection from pool. This limit is NOT from oracle db side. It is by our weblogic's configuration. It is MaxCapacity="28" for jdbc pool "JDBC-Connection-Pool-for-nGen-Application".

So, during the loadtest, itsPoolJobProcessor1-0 try to get jdbc connection from pool but failed. Then, after a while, it got timeout in waiting the jdbc connection and finally get the ORA-01013.

So, the problem is why 28 max connection is not enough.

there may be several reasons:

1) some online transaction processed longer time than normal (which hold the jdbc connection and cause tension in the jdbc connection usage).

2) some configuration changes in this nGen's release which cause the jdbc usage increased. eg. new JobProcessors...

Tested below cases, so use oracle-jdbc-10.2.0.5-java1.4.jar + oracle 10.2.0.4, the ORA-01013 error disappear.
a) oracle-jdbc-10.2.0.3-java1.4.jar + oracle 10.2.0.4 ORA-01013 occur
b) oracle-jdbc-10.2.0.5-java1.4.jar + oracle 10.2.0.4 ORA-01013 disappear
2) fyi,under study ......
http://www.coderanch.com/t/303240/JDBC/java/Oracle-driver-throws-java-sql
http://forums.oracle.com/forums/thread.jspa?threadID=1018713
3) <BEA-000627> <Reached maximum capacity of pool
http://objectmix.com/weblogic/534429-getting-bea-000627-reached-maximum-capacity-pool-but-console-saysnot.html