(Note: this list is stale.) Known Lock Order Reversals 20040820: > Also, while investigating this, witness caught a lock order reversal: > > fxp0: promiscuous mode enabled > lock order reversal > 1st 0xffffffff80653de0 bpf global lock (bpf global lock) @ /usr/src/sys/net/$ > .c:381 > 2nd 0xffffffff80851448 fxp0 (network driver) @ /usr/src/sys/dev/fxp/if_fxp.c$ > 88 > KDB: stack backtrace: > witness_checkorder() at witness_checkorder+0x654 > _mtx_lock_flags() at _mtx_lock_flags+0x4a > fxp_ioctl() at fxp_ioctl+0x6f > ifpromisc() at ifpromisc+0x98 > bpf_detachd() at bpf_detachd+0xae > bpfclose() at bpfclose+0xf8 > spec_close() at spec_close+0x1fe > vn_close() at vn_close+0x7a > vn_closefile() at vn_closefile+0x59 > fdrop_locked() at fdrop_locked+0x9f > closef() at closef+0x40 > close() at close+0xe0 > syscall() at syscall+0x4b0 > Xfast_syscall() at Xfast_syscall+0xa8 > --- syscall (6, FreeBSD ELF64, close), rip = 0x200a65640, rsp = 0x7fffffffe6c$ > rbp = 0x7fffffffe710 --- > 20040713: Jul 13 20:40:09 hippy kernel: arp_rtrequest: malloc failed Jul 13 20:40:09 hippy kernel: arplookup 10.33.40.92 failed: could not allocate l linfo Jul 13 20:40:09 hippy kernel: lock order reversal Jul 13 20:40:09 hippy kernel: 1st 0xc2909b60 rtentry (rtentry) @ net/route.c:170 Jul 13 20:40:09 hippy kernel: 2nd 0xc25a797c radix node head (radix node head) @ net/route.c:572 Jul 13 20:40:09 hippy kernel: KDB: stack backtrace: Jul 13 20:40:09 hippy kernel: kdb_backtrace(0,ffffffff,c0898650,c0898678,c082ae5 c) at kdb_backtrace+0x29 Jul 13 20:40:09 hippy kernel: witness_checkorder(c25a797c,9,c07d94f7,23c) at wit ness_checkorder+0x528 Jul 13 20:40:09 hippy kernel: _mtx_lock_flags(c25a797c,0,c07d94ee,23c,0) at _mtx _lock_flags+0x57 Jul 13 20:40:09 hippy kernel: rtexpunge(c2909b00) at rtexpunge+0x5b Jul 13 20:40:09 hippy kernel: arplookup(5c28210a,1,0,1,928210a) at arplookup+0xc 7 Jul 13 20:40:09 hippy kernel: in_arpinput(c28a7200,c28a7200,db0b3ce0,c06557a2,c2 8a7200) at in_arpinput+0x241 Jul 13 20:40:09 hippy kernel: arpintr(c28a7200) at arpintr+0xbe Jul 13 20:40:09 hippy kernel: netisr_processqueue(c08bc4d8,c22af6c0,c225de00,db0 b3d1c,c05de128) at netisr_processqueue+0x6e Jul 13 20:40:09 hippy kernel: swi_net(0) at swi_net+0x85 Jul 13 20:40:09 hippy kernel: ithread_loop(c225de00,db0b3d48,c225de00,c05ddff4,) at ithread_loop+0x134 Jul 13 20:40:09 hippy kernel: fork_exit(c05ddff4,c225de00,db0b3d48) at fork_exit+0x98 Jul 13 20:40:09 hippy kernel: fork_trampoline() at fork_trampoline+0x8 Jul 13 20:40:09 hippy kernel: --- trap 0x1, eip = 0, esp = 0xdb0b3d7c, ebp = 0 20040622: Juli reports: lock order reversal 1st 0xc17d6bd0 inp (tcpinp) @ netinet/tcp_input.c:720 2nd 0xc07576cc tcp (tcp) @ netinet/tcp_usrreq.c:616 Stack backtrace: backtrace(c06d394e,c07576cc,c06d34ec,c06d34ec,c06dbd4e) at backtrace+0x17 witness_checkorder(c07576cc,9,c06dbd4e,268,c1845690) at witness_checkorder+0x678 _mtx_lock_flags(c07576cc,0,c06dbd45,268,c1845690) at _mtx_lock_flags+0x80 tcp_usr_rcvd(c184562c,80,c06d5f2c,4b3,cbd50aa8) at tcp_usr_rcvd+0x30 soreceive(c184562c,cbd50aec,cbd50af8,cbd50af0,0) at soreceive+0xc45 nfsrv_rcv(c184562c,c1aaf180,1,19a,c17d85d0) at nfsrv_rcv+0x11f sowakeup(c184562c,c184567c,c06db32a,4e2,108) at sowakeup+0x101 tcp_input(c16cb700,14,c159d800,1,0) at tcp_input+0x1100 ip_input(c16cb700,0,c06d9133,95,c07564d8) at ip_input+0x905 netisr_processqueue(c07564d8,0,c06d9133,fd,c1524c40) at netisr_processqueue+0x8e swi_net(0,0,c06cde5d,270,0) at swi_net+0xa3 ithread_loop(c14f8d80,cbd50d48,c06cdc50,328,0) at ithread_loop+0x182 fork_exit(c0505580,c14f8d80,cbd50d48) at fork_exit+0xc2 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xcbd50d7c, ebp = 0 --- (kgdb) l *(tcp_input+0x1100) 0xc05b2ad0 is in tcp_input (../../../netinet/tcp_input.c:1250). (kgdb) l *(soreceive+0xc45) 0xc055d1c5 is in soreceive (../../../kern/uipc_socket.c:1205). 1205 SOCKBUF_LOCK(&so->so_rcv); Now believed to be fixed due to Alfred causing soreceive() not the be called on the source socket in the netisr context. This may have an as yet unmeasured performance impact, and may not be necessary for SOCK_DGRAM, only SOCK_STREAM. 20040530: FreeBSD pointyhat.freebsd.org 5.2-CURRENT FreeBSD 5.2-CURRENT #6: Mon May 31 02:28:19 GMT 2004 kris@pointyhat.freebsd.org:/usr/src/sys/i386/compile/POINTYHAT i386 lock order reversal 1st 0xc083aae0 rawcb (rawcb) @ net/raw_usrreq.c:85 2nd 0xc788ca84 so_rcv (so_rcv) @ kern/uipc_socket2.c:1072 sbappendaddr(c788ca70,c0728f78,c347ca00,0,c347ca00) at sbappendaddr+0x49 raw_input(c347ca00,efbc7a10,c0728f78,c0728f68,20011) at raw_input+0x19f rt_dispatch(c347ca00,c77d1880,c76dcb00,c76dc100,efbc7a94) at rt_dispatch+0x48 rt_missmsg(1,efbc7a58,20405,0,0) at rt_missmsg+0x53 rtalloc1(c77d1a6c,1,0,41f,c06f6413) at rtalloc1+0x220 rt_setgate(c7791900,c77d1a5c,c77d1a6c,2ee,3) at rt_setgate+0x28d rtrequest1(1,efbc7b44,efbc7b30,c77d1a80,0) at rtrequest1+0x445 route_output(c347cb00,c788ca20,80,c347cb00,0) at route_output+0x24d raw_usend(c788ca20,0,c347cb00,0,0) at raw_usend+0x73 rts_send(c788ca20,0,c347cb00,0,0) at rts_send+0x35 sosend(c788ca20,0,efbc7c80,c347cb00,0) at sosend+0x53d soo_write(c77cd990,efbc7c80,c345a600,0,c73d6d20) at soo_write+0x87 dofilewrite(c73d6d20,c77cd990,3,804ece0,80) at dofilewrite+0xfb write(c73d6d20,efbc7d14,c,6,3) at write+0x6e syscall(2f,2f,2f,804ef3c,804ed60) at syscall+0x2a0 Xint0x80_syscall() at Xint0x80_syscall+0x1f 20040523: When running with rwatson_net2, this lock order reversal occurs because routing locks are held over entry into the raw socket code in order to generate route messages. Arguably, this is a bug in the routing code, which should avoid holding locks over calls into the raw socket code for precisely this reason. Starting dhclient. lock order reversal 1st 0xc6574160 rtentry (rtentry) @ net/route.c:170 2nd 0xc09de520 rawcb_mtx (rawcb_mtx) @ net/raw_usrreq.c:85 Stack backtrace: backtrace(0,ffffffff,c09ba5b0,c09ba060,c0824e9c) at backtrace+0x12 witness_checkorder(c09de520,9,c07d31d0,55) at witness_checkorder+0x593 _mtx_lock_flags(c09de520,0,c07d31c7,55,c2288100) at _mtx_lock_flags+0x68 raw_input(c2288100,eb299a28,c082e998,c082e988,20011) at raw_input+0x27 rt_dispatch(c2288100,c65c2400,c6657b00,c6369a00,eb299aa4) at rt_dispatch+0x37 rt_missmsg(1,eb299a68,20405,0) at rt_missmsg+0x44 rtalloc1(c65c23ec,1,0,c6657760,0) at rtalloc1+0x1a6 rt_setgate(c6657700,c65c23dc,c65c23ec,80,80) at rt_setgate+0x21c rtrequest1(1,eb299b50,eb299b3c,0,0) at rtrequest1+0x356 route_output(c2288000,c6654b64,80,c2288000,1f80) at route_output+0x1fa raw_usend(c6654b64,0,c2288000,0,0,c65b2420) at raw_usend+0x6c rts_send(c6654b64,0,c2288000,0,0) at rts_send+0x1b sosend(c6654b64,0,eb299c88,c2288000,0) at sosend+0x479 soo_write(c65bd088,eb299c88,c225c600,0,c65b2420) at soo_write+0x46 dofilewrite(c65b2420,c65bd088,4,804e360,80) at dofilewrite+0xbb write(c65b2420,eb299d14,3,1,292) at write+0x3e syscall(2f,2f,2f,804e3bc,80) at syscall+0x217 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (4), eip = 0x280bb217, esp = 0xbfbfeacc, ebp = 0xbfbfeaf8 --- 20040419: pjd reports the following: lock order reversal 1st 0xc6470514 so_snd (so_snd) @ /mnt/perf/src/sys/kern/uipc_socket.c:690 2nd 0xc077206c rip (rip) @ /mnt/perf/src/sys/netinet/raw_ip.c:696 Stack backtrace: backtrace(c0620944,c077206c,c0626ff9,c0626ff9,c0627003) at backtrace+0x17 witness_checkorder(c077206c,9,c0627003,2b8,c6470514) at witness_checkorder+0x707 _mtx_lock_flags(c077206c,0,c0627003,2b8,40) at _mtx_lock_flags+0x9a rip_send(c6470438,0,c228d300,c61c68a0,0) at rip_send+0x33 sosend(c6470438,c61c68a0,e7291c4c,c228d300,0) at sosend+0x4c6 kern_sendit(c6e29bd0,3,e7291cc4,0,0) at kern_sendit+0x19a sendit(c6e29bd0,3,e7291cc4,0,804e0d4) at sendit+0x16f sendto(c6e29bd0,e7291d14,18,434,6) at sendto+0x5b syscall(2f,2f,2f,804e094,804e094) at syscall+0x271 Xint0x80_syscall() at Xint0x80_syscall+0x1d --- syscall (133), eip = 0x280dd6af, esp = 0xbfbee82c, ebp = 0xbfbee868 --- 20040408: lock order reversal 1st 0xc77bf540 so_rcv (so_rcv) @ netinet/tcp_usrreq.c:581 2nd 0xc636ae7c radix node head (radix node head) @ net/route.c:129 Stack backtrace: backtrace(0,ffffffff,c0952390,c0952570,c081e67c) at backtrace+0x12 witness_checkorder(c636ae7c,9,c07cd19b,81) at witness_checkorder+0x593 _mtx_lock_flags(c636ae7c,0,c07cd192,81,7) at _mtx_lock_flags+0x67 rtalloc1(ebbd7ae8,1,0,ebbd7ae4,c228ac40) at rtalloc1+0x61 rtalloc_ign(ebbd7ae4,0,ebbd7b10,c0662cc9,ebbd7ae4) at rtalloc_ign+0xa4 rtalloc(ebbd7ae4) at rtalloc+0xd ip_output(c228ac00,0,0,0,0) at ip_output+0x399 tcp_output(c667ea2c) at tcp_output+0xbbe tcp_usr_rcvd(c77bf4e0,0) at tcp_usr_rcvd+0xcd soreceive(c77bf4e0,0,ebbd7c88,0,0) at soreceive+0x97d soo_read(c65c0198,ebbd7c88,c77bde80,0,c68502a0) at soo_read+0x41 dofileread(c68502a0,c65c0198,3,bfbfc7a0,2000) at dofileread+0xb0 read(c68502a0,ebbd7d14,3,1,296) at read+0x3b syscall(2f,2f,2f,3,bfbfc7a0) at syscall+0x217 Xint0x80_syscall() at Xint0x80_syscall+0x1d --- syscall (3), eip = 0x2829b237, esp = 0xbfbfc78c, ebp = 0xbfbfe7b8 --- 20040331: lock order reversal 1st 0xc4d82408 so_rcv (so_rcv) @ netinet/tcp_usrreq.c:597 2nd 0xc448827c radix node head (radix node head) @ net/route.c:133 Stack backtrace: backtrace(0,ffffffff,c09385b8,c09387c0,c080c13c) at backtrace+0x12 witness_checkorder(c448827c,9,c07bc691,85) at witness_checkorder+0x593 _mtx_lock_flags(c448827c,0,c07bc688,85,7) at _mtx_lock_flags+0x67 rtalloc1(dde38ae8,1,0,dde38ae4,c199ce40) at rtalloc1+0x61 rtalloc_ign(dde38ae4,0,dde38b10,c065c031,dde38ae4) at rtalloc_ign+0xa4 rtalloc(dde38ae4) at rtalloc+0xd ip_output(c199ce00,0,0,0,0) at ip_output+0x399 tcp_output(c467aba0) at tcp_output+0xbbe tcp_usr_rcvd(c4d823a8,0) at tcp_usr_rcvd+0xcd soreceive(c4d823a8,0,dde38c88,0,0) at soreceive+0x94d soo_read(c47f65d8,dde38c88,c4a61f00,0,c46e9690) at soo_read+0x41 dofileread(c46e9690,c47f65d8,9,80813f8,1000) at dofileread+0xb0 read(c46e9690,dde38d14,3,d,200286) at read+0x3b syscall(2f,2f,2f,80568e0,9) at syscall+0x217 Xint0x80_syscall() at Xint0x80_syscall+0x1d --- syscall (3), eip = 0x29106333, esp = 0xbfbfaafc, ebp = 0xbfbfab28 --- Debugger("manual escape to debugger")