2.2.13aa6 (bugfix release II)
Andrea Arcangeli (andrea@suse.de)
Fri, 17 Dec 1999 16:34:21 +0100 (CET)
The main features of 2.2.13aa6 are:
o Support for 4Gigabyte of RAM (me and Gerhard.Wichert)
o Improved VM for high end machines with enough ram and doing
heavy I/O under high memory pressure (me)
o RAW-IO (also on bigmem) (Stephen C. Tweedie)
o updated with all showstopper/necessary bugfixes discovered into
the 2.2.x kernels over the time.
NOTE (2.2.14pre): if you don't need the 4g support and raw-io and your
machine has a workstation load (so you don't do heavy I/O) you should
ignore 2.2.13aa6 and I suggest to use 2.2.14pre14 plus my
block_dev-fs-corruption patch.
NOTE (raid): if you want to use the latest raid patches
(raid0145-19990824-2.2.11) over 2.2.13aa6 simply apply the raid patch over
2.2.13aa6 and then apply this incremental patch on the resulting kernel:
ftp://ftp.*.kernel.org/pub/linux/kernel/people/andrea/patches/v2.2/2.2.14pre11/set_blocksize-1-raid0145-19990824-2.2.11.gz
Then raid will work just fine.
Side note: I am not including the new raid code in 2.2.x because at least
raid0 is just rock solid in the stock 2.2.13 kernel and I don't want to
force people to convert the on-disk format of their raid device in order
to run 2.2.13aa6. People who wants to use raid can go with my incremental
raid fix.
Incremental description of 2.2.13aa6:
------ 2.2.13aa6 --------
diff -u --exclude version.gz 2.2.13aa5 2.2.13aa6
Only in 2.2.13aa6: block_dev-fs-corruption-1.gz
block_dev-fs-corruption-1.gz -> fixes fs corruption in the
blockdevice layer (me)
------ 2.2.13aa5 --------
diff -u --exclude version.gz 2.2.13aa4 2.2.13aa5
Only in 2.2.13aa4: buffer-races-2.2.10-A.gz
Only in 2.2.13aa5: buffer-races-2.2.13-3.gz
Only in 2.2.13aa5: ext2-1.gz
buffer-races-2.2.13-3.gz -> includes the
buffer-races-2.2.10-A.gz
features and it also fixes fs
corruption generated by hdparm or
flushb on an active filesystem and
a minor problem in sync_dev (me)
ext2-1.gz -> fixes bugs that may lead to
ext2 fs corruption or
fsync errors (me)
------ 2.2.13aa5 --------
diff -u --exclude version.gz 2.2.13aa3 2.2.13aa4
Only in 2.2.13aa4: inode-recycle-fixes.gz
Only in 2.2.13aa4: java-proc.gz
Only in 2.2.13aa4: signal-race.gz
Only in 2.2.13aa4: syncookies.gz
inode-recycle-fixes.gz -> fixes an inode leakage (me)
syncookies.gz -> fixed syncooky bug (without the
fix at the first synflood
the machine will forbid
connections to all hosts, must
check only the SYN/ACK/FIN
bit and not the data offset
and window of the incoming
packet ;). (Alan Cox)
signal-race.gz -> fixes a race in the send sig path
(David Miller)
java-proc.gz -> revertd the semantic change that
make difference between
/proc/00000$$ and /proc/$$, this
allows backwards compatibilty of
a misfeature and it _won't_ hurt
security. There's no downside
in reverting the 2.2.13 semantic
change.
------ 2.2.13aa3 --------
diff -u --exclude version.gz 2.2.13aa2 2.2.13aa3
Only in 2.2.13aa3: dcache-hashfn.gz
Only in 2.2.13aa3: fdset-fix.gz
Only in 2.2.13aa2: z-bigmem-2.2.13aa2-6.gz
Only in 2.2.13aa3: z-bigmem-2.2.13aa3-7.gz
z-bigmem-2.2.13aa3-7.gz -> fixed a obvious silly bigmem
bug that will lead
to processes killed randomly.
(all the credit goes to Leonard N.
Zubkoff)
fdset-fix.gz -> fixed a fdset bug that may lead to
memory corruption and Oopses
(credits goes to
Savochkin Andrey Vladimirovich,
I only backported the 2.3.x patch
to a four liner against 2.3.13)
dcache-hashfn.gz -> use only the dentry noise for
randomizing the dcache hashfn
(all the credit goes to David S.
Miller)
------ 2.2.13aa2 --------
SMP-scheduler-2.2.11-E.gz -> rewrote of reschedule_idle. (me)
buffer-hash.gz -> fixes lowmem box hash size. (me)
buffer-races-2.2.10-A.gz -> fixes of race condition that may lead
to bad things in invalidate_buffers()
and set_blocksize(). (me)
clear-backlog-2.gz -> fixes for a SMP race condition in
the main network backlog handling. (me)
dcache-hash.gz -> dcache hash dynamic (with my
own heuristc). (started from 2.2.13ac1
but then reimplemented by me)
free_page.gz -> cleanup of the __free_pages
interface. (me)
hashed-buffers-2.2.10.gz -> minor fix to increase the debugging
information in the right place. (me)
inode-leak-2.2.10-A.gz -> make sure to not leak memory
by allocating lots of sockets (DoS),
and let know the admin to enlarge
the max-inodes if the admin really
wants more unfreeable memory in the
icache. (me)
kupdate-sigstop-2.2.11-1.gz -> allow kupdate to be stopped via
SIGSTOP (currently it must be stopped
by setting interval to zero via
sysctl). (me)
no-swapout-2.2.10-B.gz -> avoid swapin/swapouts during heavy
I/O (strictly necessary for decent
performances on very I/O and MM loaded
servers). (me)
oom-2.2.12-I.gz -> assorted OOM fixes (deadlocks in
pagein, Alpha SIGBUS fix, avoid
sigkilling iopl() application send
a sigterm instead, avoid init
to be killed), it's the same
patch merged by Alan into 2.2.14pre2. (me)
pagecache-hash.gz -> pagecache hash dynamic (I think
it's DaveM's work, literally I took it
from 2.2.13ac1). I agree with the
heuristc used. It allocates
num_physpages buckets for the pagecache
and this basically means all the
buckets will be filled supposing a
perfect hash distribution with all the memory
allocated in the cache. (all credits
to David S. Miller)
probe-irq-2.3.14-pre2-1.gz -> avoid a pending irq to be mistaken
for a spurious irq. (me)
shrink_all_cache-2.2.10-A.gz -> make sure that big memory boxes will
shrink the cache well enough. (me)
trashing-mem-2.2.10-A.gz -> heuristic to penalize memory hogs,
the system will remains responsive
also during heavy swapout. (me)
version.gz -> set the EXTRAVERSION to aa2 ;)
wait-event-smp-races.gz -> Put the two mb() after setting the
task state as blocking and before
checking if the event is just happend
(SMP race fix). (me)
wait4-smp-race.gz -> _Critical_ SMP race fix.
Without this one liner each time you
run `ls` from bash, the bash is going
to deadlock in wait4 if you are unlucky
enough. The race is very small
but there are machine under heavy
fork load load that reproduced this
race regularly after some day of load.
The SMP race can happen only
with an SMP kernel on a SMP hardware. (me)
wakeup_bdflush-2.2.10-A.gz -> avoid deadlocking in wakeup_bdflush
(the run_task_queue() can sleep for
example while running the loop
request function). (me)
z-bigmem-2.2.13aa2-6.gz -> 4GB support on x86. (me and
Gerhard Wichert)
z-bigmem-nodebug.gz -> turn the bigmem code into production
mode.
z-bigmem-rawio-2.2.13aa2-1.gz -> rawio working even with bigmem memory
(I started with rawio from 2.2.13ac1
and SCT's 2.3.x rawio bounce buffers,
all the credits go to Stephen C.
Tweedie)
zmagic-all-blocksize.gz -> allow zmagic binaries to run
also on 4k filesystems (it's the same
that gone into 2.2.14pre2). (me)
----------------------------------------------------------------------
To go in sync with 2.2.13aa6 you can:
mkdir 2.2.13aa6
cd 2.2.13aa6
wget --retr-symlinks -A\*.gz
ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.2/2.2.13aa6/\*
cd ..
and now you'll have all the interesting patches in the directory
2.2.13aa6.
At this point rename the 2.2.13 sources to 2.2.13aa6:
mv linux-2.2.13 linux-2.2.13aa6
cd linux-2.2.13aa6
and apply all the 2.2.13aa6 patches that you previously downloaded from
the ftp site:
apply-patches.sh ../2.2.13aa6
At this point your tree will be in sync with 2.2.13aa6. Just configure
recompile and boot the new kernel.
You can find the `apply-patches.sh` bash script I written to easily apply
my kernel patches here:
ftp://ftp.*.kernel.org/pub/linux/kernel/people/andrea/tools/apply-patches/apply-patches.sh.gz
There is also a README on how to use it:
ftp://ftp.*.kernel.org/pub/linux/kernel/people/andrea/tools/apply-patches/README.gz
The 2.2.13aa6 kernel is placed here:
ftp://ftp.*.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.2/2.2.13aa6/
Have fun! ;)
Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/
From: ursus <ur...@usa.net> Subject: Re: [2.2.13aa6 (bugfix release II) ] Date: 1999/12/20 Message-ID: <fa.l5tasev.1u742a6@ifi.uio.no>#1/1 X-Deja-AN: 563124487 Original-Date: 20 Dec 99 14:51:32 EST Sender: owner-linux-ker...@vger.rutgers.edu Content-Transfer-Encoding: 8BIT Original-Message-ID: <19991220195132.10068.qmail@nw179.netaddress.usa.net> To: Andrea Arcangeli <and...@suse.de> X-Priority: 1 Content-Type: text/plain; charset=US-ASCII X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list Read-Receipt: ur...@usa.net X-MSMail-Priority: High Disposition-Notification-To: ur...@usa.net Mime-Version: 1.0 Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu In newsgroup fa.linux.kernel, Andrea Arcangeli wrote: > Date: Fri, 17 Dec 1999 16:34:21 +0100 (CET) > From: Andrea Arcangeli <and...@suse.de> > Subject: 2.2.13aa6 (bugfix release II) > > [...] > The main features of 2.2.13aa6 are: > > o Support for 4Gigabyte of RAM (me and Gerhard.Wichert) > o Improved VM for high end machines with enough ram and doing > heavy I/O under high memory pressure (me) > o RAW-IO (also on bigmem) (Stephen C. Tweedie) > > o updated with all showstopper/necessary bugfixes discovered into > the 2.2.x kernels over the time. > Andrea: Thanks for the updated 2.2.13aa6 patchset, especially that it works with the raid-0.90 patches cleanly! I've been using Alan Cox's 2.2.13ac3 patches for the raid-0.90 support, but really wanted to run with your SMP scheduling changes, since they would seem to help performance/stability with my application (high-load webserver on dual-PIII machine). Also I was getting errors regarding "Out of memory" which you have a couple of patches for in aa6 ... I upgraded a cluster of servers (Compaq 6400R, 2 x PIII-500) from 2.2.13ac3 to 2.2.13aa6+raid-0.90 (and the incremental "set_blocksize" patch you kindly provided) and Don Becker's eepro.c 1.09l (not sure if this is latest?) in hopes I can finally have a really stable setup ... these had been running well for about 12 hours, but I just had one of the servers crash with the following error (seen before under 2.2.13ac3): wait_on_bh, CPU 3: (this is the first processor) irq: 0 [0 0] bh: 1 [0 0] <[8010b39d]> <[80150daa]> <[80150d46]> <[8012912b]> \ <[8012a367]> <[801291a6]> <[8012921f]> <[801092ac]> I tried to correlate the registers above with System.map: 8010b360 T synchronize_bh 8010b3b0 T synchronize_irq 80150d20 t sock_close 80150d5c t sock_fasync 8012910c T __fput 80129154 T filp_close 8012a350 T fput 8012a398 T put_filp 80129154 T filp_close 801291b0 T sys_close 801291b0 T sys_close 80129238 T sys_vhangup 80109278 T system_call 801092b0 T ret_from_sys_call If I press ALT+SysRq+P, the EIP shows "0010:[<80166671>]" which appears to be related to functions (from System.map): 80166660 T tcp_send_delayed_ack 801666b4 T tcp_send_ack In some earlier posts I read that "wait_on_bh" means that the system is waiting on the bottom half (SMP-specific), so I've edited my /etc/lilo.conf to add "nosmp noapic", and I'll see if the servers run stable w/o SMP ... this isn't a real solution of course. Any help/pointers/patches would be greatly appreciated. In an earlier post I mentioned this is part of a larger project to upgrade about 100 webservers based on 2.0.36 kernel to 2.2.13+ ... the overall load is 1Billion hits per day currently. This would be a yet another testament to Linux's viability in the enterprise environment, assuming I can nail down this SMP problem :) PS: in your directory on the ftp.*.kernel.org mirrors, I see a patch regarding bh_latency for 2.2.14pre; does this address the above "wait_on_bh" problem? Thanks in advance -- ur...@usa.net ____________________________________________________________________ Get free email and a permanent address at http://www.netaddress.com/?N=1 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: Andrea Arcangeli <and...@suse.de> Subject: Re: [2.2.13aa6 (bugfix release II) ] Date: 1999/12/21 Message-ID: <fa.m8ag9hv.vi2dpo@ifi.uio.no>#1/1 X-Deja-AN: 563335229 Original-Date: Tue, 21 Dec 1999 10:42:38 +0100 (CET) Sender: owner-linux-ker...@vger.rutgers.edu Original-Message-ID: <Pine.LNX.4.21.9912211015000.24670-100000@Fibonacci.suse.de> References: <fa.l5tasev.1u742a6@ifi.uio.no> X-PGP-Key-URL: http://e-mind.com/~andrea/aa.asc To: ursus <ur...@usa.net> X-Authentication-Warning: Fibonacci.suse.de: andrea owned process doing -bs Content-Type: TEXT/PLAIN; charset=US-ASCII X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list MIME-Version: 1.0 X-GnuPG-Key-URL: http://e-mind.com/~andrea/aa.gnupg.asc Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu On 20 Dec 1999, ursus wrote: >I upgraded a cluster of servers (Compaq 6400R, 2 x PIII-500) >from 2.2.13ac3 to 2.2.13aa6+raid-0.90 (and the incremental >"set_blocksize" patch you kindly provided) and Don Becker's That's a fine kernel ;) >[..] I just had one of the servers >crash with the following error (seen before under 2.2.13ac3): I changed nothing in my aa patches related to the problem you have, so it's normal you get it as in the 2.2.13ac3 kernel. Your report gives interesting info, thanks. >to add "nosmp noapic", and I'll see if the servers >run stable w/o SMP ... this isn't a real solution I bet it will be rock solid in UP. This looks like a genuine SMP race (of course trusting it's not an hardware issue). >assuming I can nail down this SMP problem :) We'll nail it down ;). > I see a patch regarding bh_latency for 2.2.14pre; > does this address the above "wait_on_bh" problem? It's won't help you, it's performance stuff (and it's not complete yet). Andrea - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: ur...@usa.net Subject: Re: [2.2.13aa6 (bugfix release II)] Date: 1999/12/23 Message-ID: <83tdcd$hr8$1@nnrp1.deja.com>#1/1 X-Deja-AN: 564153809 To: and...@suse.de X-Http-User-Agent: Mozilla (X11; I; Linux 2.0.32 i586) X-Http-Proxy: 1.0 x38.deja.com:80 (Squid/1.1.22) for client 199.95.209.163 Organization: Deja.com - Before you buy. X-Article-Creation-Date: Thu Dec 23 14:59:58 1999 GMT X-MyDeja-Info: XMYDJUIDray450 Reply-To: ur...@usa.net Newsgroups: fa.linux.kernel In article <fa.m8ag9hv.vi2...@ifi.uio.no>, Andrea Arcangeli <and...@suse.de> wrote: > I changed nothing in my aa patches related to the problem > you have, so it's normal you get it as in the 2.2.13ac3 kernel. Andrea: This explains why I'm still having the hangs :( > I bet it will be rock solid in UP. > This looks like a genuine SMP race I've been testing the same servers with "nosmp noapic" appended to the bootprompt (via LILO) without success; while I don't see the wait_on_bh crash anymore, instead the system just hangs without any errors whatsoever, I'll see a "normal" login prompt at the console except no characters are echoed and only SysRq [partially] works. I can't successfully Sync, Unmount via SysRq but reBoot does work. Also nothing is logged in /var/log/messages regarding the crash. I also recompiled 2.2.13aa6 for true UniProcessor mode and ran the UP kernel on the same servers for 2 days, with the same errorless crashes under heavy network load. I haven't been able to get your IKD patchset into this kernel (and boot successfully, that is). If I press ALT+SysRq+P after one of these crashes, the EIP always points to an address which maps to "timer_bh" (according to System.map) Is this the same timer_bh problem William Montgomory was discussing with you in another recent thread? > We'll nail it down ;). In article <fa.m8ag9hv.vi2...@ifi.uio.no>, Andrea Arcangeli <and...@suse.de> wrote: > I changed nothing in my aa patches related to the problem > you have, so it's normal you get it as in the 2.2.13ac3 kernel. Andrea: This explains why I'm still having the hangs :( > I bet it will be rock solid in UP. > This looks like a genuine SMP race I've been testing the same servers with "nosmp noapic" appended to the bootprompt (via LILO) without success; while I don't see the wait_on_bh crash anymore, instead the system just hangs without any errors whatsoever, I'll see a "normal" login prompt at the console except no characters are echoed and only SysRq [partially] works. I can't successfully Sync, Unmount via SysRq but reBoot does work. Also nothing is logged in /var/log/messages regarding the crash. I also recompiled 2.2.13aa6 for true UniProcessor mode and ran the UP kernel on the same servers for 2 days, with the same errorless crashes under heavy network load. I haven't been able to get your IKD patchset into this kernel (and boot successfully, that is). If I press ALT+SysRq+P after one of these crashes, the EIP always points to an address which maps to "timer_bh" (according to System.map) Is this the same timer_bh problem William Montgomory was discussing with you in another recent thread? > We'll nail it down ;). Thanks for your and the list-members' assistance ... Please let me know if I can help in any way, whether to test patches, provide crash traces, etc. -- ur...@usa.net Sent via Deja.com http://www.deja.com/ Before you buy.
From: Andrea Arcangeli <and...@suse.de> Subject: Re: [2.2.13aa6 (bugfix release II) ] Date: 2000/01/09 Message-ID: <fa.jrefbqv.a0ulib@ifi.uio.no>#1/1 X-Deja-AN: 570505709 Original-Date: Sun, 9 Jan 2000 21:43:01 +0100 (CET) Sender: owner-linux-ker...@vger.rutgers.edu Original-Message-ID: <Pine.LNX.4.21.0001092136450.11394-100000@alpha.random> References: <fa.l5tasev.1u742a6@ifi.uio.no> X-PGP-Key-URL: http://e-mind.com/~andrea/aa.asc To: ursus <ur...@usa.net> X-Sender: and...@alpha.random Content-Type: TEXT/PLAIN; charset=US-ASCII X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list MIME-Version: 1.0 X-GnuPG-Key-URL: http://e-mind.com/~andrea/aa.gnupg.asc Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu On 20 Dec 1999, ursus wrote: >I upgraded a cluster of servers (Compaq 6400R, 2 x PIII-500) >from 2.2.13ac3 to 2.2.13aa6+raid-0.90 (and the incremental >"set_blocksize" patch you kindly provided) and Don Becker's >eepro.c 1.09l (not sure if this is latest?) in hopes I can >finally have a really stable setup ... these had been running >well for about 12 hours, but I just had one of the servers >crash with the following error (seen before under 2.2.13ac3): > > wait_on_bh, CPU 3: (this is the first processor) > irq: 0 [0 0] > bh: 1 [0 0] > <[8010b39d]> <[80150daa]> <[80150d46]> <[8012912b]> \ > <[8012a367]> <[801291a6]> <[8012921f]> <[801092ac]> > >I tried to correlate the registers above with System.map: > > 8010b360 T synchronize_bh > 8010b3b0 T synchronize_irq > > 80150d20 t sock_close > 80150d5c t sock_fasync > > 8012910c T __fput > 80129154 T filp_close > > 8012a350 T fput > 8012a398 T put_filp > > 80129154 T filp_close > 801291b0 T sys_close > > 801291b0 T sys_close > 80129238 T sys_vhangup > > 80109278 T system_call > 801092b0 T ret_from_sys_call > >If I press ALT+SysRq+P, the EIP shows "0010:[<80166671>]" >which appears to be related to functions (from System.map): > > 80166660 T tcp_send_delayed_ack > 801666b4 T tcp_send_ack > >In some earlier posts I read that "wait_on_bh" >means that the system is waiting on the bottom half >(SMP-specific), so I've edited my /etc/lilo.conf >to add "nosmp noapic", and I'll see if the servers >run stable w/o SMP ... this isn't a real solution >of course. > >Any help/pointers/patches would be greatly appreciated. I think I spotted and fixed the bug that is soft-deadlocking your 2.2.x compaq cluster (all seems to make sense :). Could you try the below patch against 2.2.14 (or 2.2.14aa1 or 2.2.13 or 2.2.13aa6)? --- 2.2.14/net/ipv4/tcp_output.c.~1~ Fri Jan 7 18:19:25 2000 +++ 2.2.14/net/ipv4/tcp_output.c Sun Jan 9 21:32:04 2000 @@ -1004,7 +1004,7 @@ unsigned long timeout; /* Stay within the limit we were given */ - timeout = tp->ato; + timeout = (tp->ato << 1) >> 1; if (timeout > max_timeout) timeout = max_timeout; timeout += jiffies; I uploaded the above patch here too: ftp://ftp.*.kernel.org/pub/linux/kernel/people/andrea/patches/v2.2/2.2.14/delack-timer-1.gz Have fun! Andrea - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: "David S. Miller" <da...@redhat.com> Subject: Re: [2.2.13aa6 (bugfix release II) ] Date: 2000/01/09 Message-ID: <fa.hesm80v.157e704@ifi.uio.no>#1/1 X-Deja-AN: 570541496 Original-Date: Sun, 9 Jan 2000 15:36:09 -0800 Sender: owner-linux-ker...@vger.rutgers.edu Original-Message-Id: <200001092336.PAA11026@pizda.ninka.net> References: <fa.jrefbqv.a0ulib@ifi.uio.no> To: and...@suse.de Original-References: <Pine.LNX.4.21.0001092136450.11394-100...@alpha.random> X-Authentication-Warning: pizda.ninka.net: davem set sender to da...@redhat.com using -f X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu Date: Sun, 9 Jan 2000 21:43:01 +0100 (CET) From: Andrea Arcangeli <and...@suse.de> I think I spotted and fixed the bug that is soft-deadlocking your 2.2.x compaq cluster (all seems to make sense :). Could you try the below patch against 2.2.14 (or 2.2.14aa1 or 2.2.13 or 2.2.13aa6)? Wrong, all callers of tcp_send_delayed_ack _guarentee_ that the quickack bit is clear. Your patch does nothing, put an assert there if you don't believe me. Later, David S. Miller da...@redhat.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: Andrea Arcangeli <and...@suse.de>
Subject: Re: [2.2.13aa6 (bugfix release II) ]
Date: 2000/01/10
Message-ID: <fa.jof3chv.a0elqb@ifi.uio.no>#1/1
X-Deja-AN: 570872471
Original-Date: Mon, 10 Jan 2000 16:11:28 +0100 (CET)
Sender: owner-linux-ker...@vger.rutgers.edu
Original-Message-ID: <Pine.LNX.4.21.0001100246170.18403-100000@alpha.random>
References: <fa.hesm80v.157e704@ifi.uio.no>
X-PGP-Key-URL: http://e-mind.com/~andrea/aa.asc
To: "David S. Miller" <da...@redhat.com>
X-Sender: and...@alpha.random
Content-Type: TEXT/PLAIN; charset=US-ASCII
X-Orcpt: rfc822;linux-kernel-outgoing-dig
Organization: Internet mailing list
MIME-Version: 1.0
X-GnuPG-Key-URL: http://e-mind.com/~andrea/aa.gnupg.asc
Newsgroups: fa.linux.kernel
X-Loop: majord...@vger.rutgers.edu
On Sun, 9 Jan 2000, David S. Miller wrote:
> Date: Sun, 9 Jan 2000 21:43:01 +0100 (CET)
> From: Andrea Arcangeli <and...@suse.de>
>
> I think I spotted and fixed the bug that is soft-deadlocking your
> 2.2.x compaq cluster (all seems to make sense :). Could you try the
> below patch against 2.2.14 (or 2.2.14aa1 or 2.2.13 or 2.2.13aa6)?
>
>Wrong, all callers of tcp_send_delayed_ack _guarentee_ that the
>quickack bit is clear. Your patch does nothing, put an assert
tcp_delack_timer doesn't really guarantee that, see:
void tcp_delack_timer(unsigned long data)
{
struct sock *sk = (struct sock*)data;
if(!sk->zapped &&
sk->tp_pinfo.af_tcp.delayed_acks &&
sk->state != TCP_CLOSE) {
/* If socket is currently locked, defer the ACK. */
if (!atomic_read(&sk->sock_readers))
tcp_send_ack(sk);
else
tcp_send_delayed_ack(&(sk->tp_pinfo.af_tcp), HZ/10);
}
}
You should as well guarantee by design that none timer is pending after
you turn on the quickack bit and before dropping the bh or sock lock. It
seems to me you are guaranteeing that by always sending an ack on the wire
after you set up the quickack bit but it's not trivial to prove and right
now the only explanation for the deadlock reported by urban and that other
people is experiencing is that a delack timer triggers while delayed_acks
is > 0 and the quickack bit is set.
If the quickack bit is set while calling tcp_send_delayed_ack the kernel
will lockup immediatly in a way that matches the reports from ursus.
The reason for the deadlock is that the expired field of the timer will be
set in the past and so the timer will reinsert inself in the first heap
slot and so it will continue to reinsert and rexecute it in an infinite
loop -> soft deadlock. I'd like to also make the timer code robust against
these kind of subsystem bugs later but actually I am only focused to fix
the offending code in TCP.
I have to admit that I can't yet see exactly the path that sets the
quickack bit without sending data on the wire but you agree with me that
the tcp_send_delayed_ack function is not interested about the quickack bit
and it's interested only about the real "ato" information, so my patch is
obviously correct and in the worst case it won't change anything. I
believe it will as well fix the lockup fine, and that it's the right
approch to avoid these kind of subtle mistakes. It makes more sense than
destroying the quickack information before reinserting the the delack
timer from the delack timer, no?
BTW, while reading the code I found a lockup-unrelated bug in the delack
handling:
--- 2.2.14/net/ipv4/tcp_input.c Fri Jan 7 18:19:25 2000
+++ /tmp/tcp_input.c Mon Jan 10 03:41:10 2000
@@ -1428,6 +1428,7 @@
if (!after(TCP_SKB_CB(skb)->end_seq, tp->rcv_nxt)) {
/* A retransmit, 2nd most common case. Force an imediate ack. */
SOCK_DEBUG(sk, "retransmit received: seq %X\n", TCP_SKB_CB(skb)->seq);
+ tp->delayed_acks++;
tcp_enter_quickack_mode(tp);
kfree_skb(skb);
return;
Anyway the above fix is not interesting for real world because it seems
impossible to me to reach such path in the TCP code (so basically such
check is useless but after all I like it for completeness of the
function). This because we call tcp_data only when we know the packet is
in our receive window (otherwise we force an ack by hand prior calling
tcp_queue).
And really tp->delayed_acks is meaningless as far I can tell and the right
thing to do is to remove the delayed_acks field completly (this must be
done at least in 2.3.x). Removing it will avoid wasting time in the TCP
code and will decrease half of the the delacks code braindamage :).
Ursus, please apply also this patch on the top of my fix in the previous
email as for David correct suggestion of putting an assert there. If
you'll see a printk with the below patch applyed we'll have the proof my
theory about the source of your deadlocks is correct and that my fix
made the difference for you. Without the below patch applyed you could
think you are not deadlocking anymore because of luck :).
--- 2.2.14/net/ipv4/tcp_timer.c Fri Jan 7 18:19:25 2000
+++ /tmp/tcp_timer.c Mon Jan 10 16:02:14 2000
@@ -173,7 +173,12 @@
if (!atomic_read(&sk->sock_readers))
tcp_send_ack(sk);
else
+ {
+ struct tcp_opt *tp = &(sk->tp_pinfo.af_tcp);
+ if (tcp_in_quickack_mode(tp))
+ printk(KERN_ERR "quickack bit set!!!!\n");
tcp_send_delayed_ack(&(sk->tp_pinfo.af_tcp), HZ/10);
+ }
}
}
If my fix doesn't fix the deadlock completly I have really no other
rasonable ideas on what can be going wrong right now. Thinking thinking...
Comments?
Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/
From: "David S. Miller" <da...@redhat.com> Subject: Re: [2.2.13aa6 (bugfix release II) ] Date: 2000/01/11 Message-ID: <fa.hfcqagv.13m24g2@ifi.uio.no>#1/1 X-Deja-AN: 570965301 Original-Date: Mon, 10 Jan 2000 11:43:40 -0800 Sender: owner-linux-ker...@vger.rutgers.edu Original-Message-Id: <200001101943.LAA11888@pizda.ninka.net> References: <fa.jof3chv.a0elqb@ifi.uio.no> To: and...@suse.de Original-References: <Pine.LNX.4.21.0001100246170.18403-100...@alpha.random> X-Authentication-Warning: pizda.ninka.net: davem set sender to da...@redhat.com using -f X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu Date: Mon, 10 Jan 2000 16:11:28 +0100 (CET) From: Andrea Arcangeli <and...@suse.de> tcp_delack_timer doesn't really guarantee that, see: Ok, thanks for catching this. If the quickack bit is set while calling tcp_send_delayed_ack the kernel will lockup immediatly in a way that matches the reports from ursus. How? Always in such a case, timeout > max_timeout because this bit is set and the values are unsigned. The reason for the deadlock is that the expired field of the timer will be set in the past and so the timer will reinsert inself in the first heap slot and so it will continue to reinsert and rexecute it in an infinite loop -> soft deadlock. Not if the timeout>max_timeout test passes, which I think it will. I'd like to also make the timer code robust against these kind of subsystem bugs later but actually I am only focused to fix the offending code in TCP. Agreed, so I want your fix to go in anyways. But I do want to discuss where the error comes from and why the timeout>max_timeout test does not prevent it. And really tp->delayed_acks is meaningless as far I can tell and the right thing to do is to remove the delayed_acks field completly (this must be done at least in 2.3.x). Removing it will avoid wasting time in the TCP code and will decrease half of the the delacks code braindamage :). It's already done in the patch sets I've been feeding Linus for 2.3.x It has died in our sources, it will be no more :-) Later, David S. Miller da...@redhat.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: ursus <ur...@usa.net> Subject: Re: [Re: [2.2.13aa6 (bugfix release II) ]] Date: 2000/01/10 Message-ID: <fa.nsvi2rv.1e4401n@ifi.uio.no>#1/1 X-Deja-AN: 570902240 Original-Date: 10 Jan 00 13:30:13 EST Sender: owner-linux-ker...@vger.rutgers.edu Content-Transfer-Encoding: 8BIT Original-Message-ID: <20000110183013.13273.qmail@nwcst314.netaddress.usa.net> To: Andrea Arcangeli <and...@suse.de>, "David S. Miller" <da...@redhat.com> Content-Type: text/plain; charset=US-ASCII X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list Mime-Version: 1.0 Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu Andrea Arcangeli <and...@suse.de> wrote: > please apply also this patch on the top of my fix > in the previous email as for David [Miller's] correct > suggestion of putting an assert there. If you see > a printk with the below patch applyed, we'll have > [proof that] my theory about the source of your deadlocks > is correct and that my fix made the difference for you. ok, I'll apply the previous "delack-timer-1" patch, as well as the one below. However, can you upload the below patch to the ftp.*.kernel.org mirrors also, just so I can ensure the spacings and such are correct. Thanks. > --- 2.2.14/net/ipv4/tcp_timer.c Fri Jan 7 18:19:25 2000 As a side note, the machines have still have not crashed, so almost certainly the TCP_DELAY_ACK bug is the culprit, at least in UniProcessor case. Going to try using SMP kernel again after I'm sure these patches do the job. Thanks for your help ... -- RW <ur...@usa.net> ____________________________________________________________________ Get free email and a permanent address at http://www.netaddress.com/?N=1 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: Andrea Arcangeli <and...@suse.de> Subject: Re: [Re: [2.2.13aa6 (bugfix release II) ]] Date: 2000/01/10 Message-ID: <fa.inbme8v.1a58kjb@ifi.uio.no>#1/1 X-Deja-AN: 570914675 Original-Date: Mon, 10 Jan 2000 19:53:05 +0100 (CET) Sender: owner-linux-ker...@vger.rutgers.edu Original-Message-ID: <Pine.LNX.4.21.0001101945490.4259-100000@alpha.random> References: <fa.nsvi2rv.1e4401n@ifi.uio.no> X-PGP-Key-URL: http://e-mind.com/~andrea/aa.asc To: ursus <ur...@usa.net> X-Sender: and...@alpha.random Content-Type: TEXT/PLAIN; charset=US-ASCII X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list MIME-Version: 1.0 X-GnuPG-Key-URL: http://e-mind.com/~andrea/aa.gnupg.asc Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu On 10 Jan 2000, ursus wrote: >as well as the one below. However, can you upload >the below patch to the ftp.*.kernel.org mirrors also, ok, I put it here: ftp://ftp.*.kernel.org/pub/linux/kernel/people/andrea/patches/v2.2/2.2.14/delack-assert-1.gz >at least in UniProcessor case. Going to try using SMP >kernel again after I'm sure these patches do the job. Fine. BTW, assuming TCP does its locking right (either doing lock_sock or running in atomic bh context) SMP/UP shouldn't matter. And if TCP is missing a lock_sock a race can trigger also in UP. So hopefully if we get it fixed on UP we should be fine on SMP too later. Andrea - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: Andrea Arcangeli <and...@suse.de>
Subject: timer_bh robusteness fix against potential deadlocks
Date: 2000/01/12
Message-ID: <fa.hpj3vlv.752u12@ifi.uio.no>#1/1
X-Deja-AN: 571586341
Original-Date: Wed, 12 Jan 2000 02:00:02 +0100 (CET)
Sender: owner-linux-ker...@vger.rutgers.edu
Original-Message-ID: <Pine.LNX.4.21.0001120135430.312-100000@alpha.random>
References: <fa.nsvi2rv.1e4401n@ifi.uio.no>
X-PGP-Key-URL: http://e-mind.com/~andrea/aa.asc
To: ursus <ur...@usa.net>
X-Sender: and...@alpha.random
Content-Type: TEXT/PLAIN; charset=US-ASCII
X-Orcpt: rfc822;linux-kernel-outgoing-dig
Organization: Internet mailing list
MIME-Version: 1.0
X-GnuPG-Key-URL: http://e-mind.com/~andrea/aa.gnupg.asc
Newsgroups: fa.linux.kernel
X-Loop: majord...@vger.rutgers.edu
I fixed the timer code to be robust against the bad scenario I discovered
in the last days. The bad secnario consists in a timer that reinsert
itself with an expire <= jiffies (or more precisely < timer_jiffies).
In the current 2.2.x and 2.3.x this scenario will lead in a plain
deadlock.
I verifyed the correctness the patch in an userspace simulation.
Ursus could you please check out if you can still deadlock your machine
with this patch against 2.2.14 applyed on the top of your current tree?
--- 2.2.14/kernel/sched.c Wed Jan 5 14:16:56 2000
+++ /tmp/sched.c Wed Jan 12 00:45:15 2000
@@ -535,6 +535,15 @@
/* can happen if you add a timer with expires == jiffies,
* or you set a timer to go off in the past
*/
+ if ((signed long) idx < -50)
+ /* Nobody should set a timer so insanely in the past or
+ waiting so many timer interrupts between reading
+ jiffies and calling the timer code. The timer code
+ is completly robust against this condition but
+ a printk may let us know about bugs in the
+ caller we might not notice otherwise. */
+ printk(KERN_WARNING
+ "timer inserted in the past, idx = %ld\n", idx);
insert_timer(timer, tv1.vec, tv1.index);
} else if (idx <= 0xffffffffUL) {
int i = (expires >> (TVR_BITS + 3 * TVN_BITS)) & TVN_MASK;
@@ -1124,10 +1133,31 @@
tv->index = (tv->index + 1) & TVN_MASK;
}
+/* defer current timers to the next pass */
+static void cascade_current_timers(void)
+{
+ struct timer_list * timer;
+ int index = tv1.index;
+
+ timer = tv1.vec[index];
+ tv1.index = (tv1.index + 1) & TVR_MASK;
+
+ while (timer)
+ {
+ struct timer_list *tmp = timer;
+ timer = timer->next;
+ insert_timer(tmp, tv1.vec, tv1.index);
+ }
+ tv1.vec[index] = NULL;
+}
+
static inline void run_timer_list(void)
{
+ long passes;
+
spin_lock_irq(&timerlist_lock);
- while ((long)(jiffies - timer_jiffies) >= 0) {
+ passes = jiffies - timer_jiffies;
+ while (passes-- >= 0) {
struct timer_list *timer;
if (!tv1.index) {
int n = 1;
@@ -1135,17 +1165,21 @@
cascade_timers(tvecs[n]);
} while (tvecs[n]->index == 1 && ++n < NOOF_TVECS);
}
- while ((timer = tv1.vec[tv1.index])) {
+ timer = tv1.vec[tv1.index];
+ tv1.vec[tv1.index] = 0;
+ while (timer) {
void (*fn)(unsigned long) = timer->function;
unsigned long data = timer->data;
- detach_timer(timer);
- timer->next = timer->prev = NULL;
+ struct timer_list * tmp = timer;
+ timer = timer->next;
+ detach_timer(tmp);
+ tmp->next = tmp->prev = NULL;
spin_unlock_irq(&timerlist_lock);
fn(data);
spin_lock_irq(&timerlist_lock);
}
++timer_jiffies;
- tv1.index = (tv1.index + 1) & TVR_MASK;
+ cascade_current_timers();
}
spin_unlock_irq(&timerlist_lock);
}
The same patch will apply cleanly also to 2.3.38 by specifying as file to
patch linux/kernel/timer.c .
Or you can download the patch from here:
ftp://ftp.*.kernel.org/pub/linux/kernel/people/andrea/patches/v2.2/2.2.14/timer_bh-deadlock-1.gz
ftp://ftp.*.kernel.org/pub/linux/kernel/people/andrea/patches/v2.3/2.3.38/timer_bh-deadlock-1.gz
Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/
From: Andrea Arcangeli <and...@suse.de> Subject: Re: timer_bh robusteness fix against potential deadlocks Date: 2000/01/12 Message-ID: <fa.hs37vlv.2l2u1e@ifi.uio.no>#1/1 X-Deja-AN: 571572295 Original-Date: Wed, 12 Jan 2000 02:49:56 +0100 (CET) Sender: owner-linux-ker...@vger.rutgers.edu Original-Message-ID: <Pine.LNX.4.21.0001120239430.822-100000@alpha.random> References: <fa.hpj3vlv.752u12@ifi.uio.no> X-PGP-Key-URL: http://e-mind.com/~andrea/aa.asc To: ursus <ur...@usa.net> X-Sender: and...@alpha.random Content-Type: TEXT/PLAIN; charset=US-ASCII X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list MIME-Version: 1.0 X-GnuPG-Key-URL: http://e-mind.com/~andrea/aa.gnupg.asc Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu On Wed, 12 Jan 2000, Andrea Arcangeli wrote: >I fixed the timer code to be robust against the bad scenario I discovered >in the last days. The bad secnario consists in a timer that reinsert >itself with an expire <= jiffies (or more precisely < timer_jiffies). As TyTso suggested the problem could be ato to be zero. I checked and I finally also spotted the offending bug that was causing the above condition to happen (second chunk of the patch): --- 2.2.14-tcp/net/ipv4/tcp_output.c.~1~ Fri Jan 7 18:19:25 2000 +++ 2.2.14-tcp/net/ipv4/tcp_output.c Wed Jan 12 02:47:32 2000 @@ -1004,7 +1004,7 @@ unsigned long timeout; /* Stay within the limit we were given */ - timeout = tp->ato; + timeout = (tp->ato << 1) >> 1; if (timeout > max_timeout) timeout = max_timeout; timeout += jiffies; @@ -1044,6 +1044,8 @@ */ if(tcp_in_quickack_mode(tp)) tcp_exit_quickack_mode(tp); + if (!tp->ato) + tp->ato = tp->rto; tcp_send_delayed_ack(tp, HZ/2); return; } An incoming synack doesn't carry any data into the packet so the tcp_delack_estimator gets not recalled from tcp_ack, and the ato stays zero. Then tcp_send_ack (the one we send to put the other end in enstablished state) goes oom and queue the delack timer while ato is still zero. Then the timer gets reinserted in the current queue from run_timer_list and boom! The fact an oom condition was necessary to trigger the bug, perfectly explains why it wasn't reproducible in most machines. ftp://ftp.*.kernel.org/pub/linux/kernel/people/andrea/patches/v2.2/2.2.14/delack-timer-2.gz Andrea - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: "David S. Miller" <da...@redhat.com> Subject: Re: timer_bh robusteness fix against potential deadlocks Date: 2000/01/12 Message-ID: <fa.hacs8ov.17nc6oe@ifi.uio.no>#1/1 X-Deja-AN: 571586340 Original-Date: Tue, 11 Jan 2000 18:40:23 -0800 Sender: owner-linux-ker...@vger.rutgers.edu Original-Message-Id: <200001120240.SAA02150@pizda.ninka.net> References: <fa.hs37vlv.2l2u1e@ifi.uio.no> To: and...@suse.de Original-References: <Pine.LNX.4.21.0001120239430.822-100...@alpha.random> X-Authentication-Warning: pizda.ninka.net: davem set sender to da...@redhat.com using -f X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu Date: Wed, 12 Jan 2000 02:49:56 +0100 (CET) From: Andrea Arcangeli <and...@suse.de> @@ -1044,6 +1044,8 @@ */ if(tcp_in_quickack_mode(tp)) tcp_exit_quickack_mode(tp); + if (!tp->ato) + tp->ato = tp->rto; tcp_send_delayed_ack(tp, HZ/2); return; } Yep, I bet this is it. Good spotting. Both of these fixes to tcp_output.c are in my tree now. Later, David S. Miller da...@redhat.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: ty...@valinux.com
Subject: Re: timer_bh robusteness fix against potential deadlocks
Date: 2000/01/14
Message-ID: <fa.ikttrov.2ks7bu@ifi.uio.no>#1/1
X-Deja-AN: 572547617
Original-Date: Wed, 12 Jan 2000 11:23:45 -0800
Sender: owner-linux-ker...@vger.rutgers.edu
Original-Message-Id: <E128TMv-0007md-00@dcl.su.varesearch.com>
References: <fa.ikaud7v.1b50lj0@ifi.uio.no>
To: and...@suse.de
X-Orcpt: rfc822;linux-kernel-outgoing-dig
Organization: Internet mailing list
Phone: (781) 391-3464
Newsgroups: fa.linux.kernel
X-Loop: majord...@vger.rutgers.edu
Date: Wed, 12 Jan 2000 20:12:54 +0100 (CET)
From: Andrea Arcangeli <and...@suse.de>
>It adds one conditional inside a rare 'if' case, so it's not a
>performance issue, and it means that the next time something like this
>happens, the machine will cleanly panic, and leave a very easy to
>understand indication of what went wrong.
I don't like a panic for a thing that we can recover gracefully and
allowing the user to also see the message even if he was running under X 8).
Fine,so make it set a standard timeout and do a printk instead. This is
a "never can happen" situation, right?
if (!timeout) {
timeout = tp->rto;
if (!timeout) {
printk("Bugcheck: tcp_send_delayed_ack ato and rto are 0");
timeout = HZ/50;
}
- Ted
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/
From: ty...@valinux.com
Subject: Re: timer_bh robusteness fix against potential deadlocks
Date: 2000/01/14
Message-ID: <fa.iib9h8v.3609ru@ifi.uio.no>#1/1
X-Deja-AN: 572550514
Original-Date: Wed, 12 Jan 2000 07:59:35 -0800
Sender: owner-linux-ker...@vger.rutgers.edu
Original-Message-Id: <E128QBL-0007kE-00@dcl.su.varesearch.com>
References: <fa.ilr2dov.18l4l35@ifi.uio.no>
To: and...@suse.de
X-Orcpt: rfc822;linux-kernel-outgoing-dig
Organization: Internet mailing list
Phone: (781) 391-3464
Newsgroups: fa.linux.kernel
X-Loop: majord...@vger.rutgers.edu
Date: Wed, 12 Jan 2000 04:02:35 +0100 (CET)
From: Andrea Arcangeli <and...@suse.de>
Ok, moving the check inside the tcp_send_delayed_ack it's fine for me.
But there's to say that the only other way the delack timer could be
posted is via __tcp_ack_snd_check. And if __tcp_ack_snd_check is using
tcp_send_delayed_ack instead of tcp_send_ack before in order packets with
data are arrived (so before the ato is been initalized to something
different than zero) it probably means there's a genuine bug in tcp.
True; but my paranoia says that even if there isn't a problem *now*,
there may be later. Which is why why I'll suggest one more change to
your patch:
if (!timeout) {
timeout = tp->rto;
if (!timeout)
panic("Bugcheck: tcp_send_delayed_ack ato and rto are 0");
}
It adds one conditional inside a rare 'if' case, so it's not a
performance issue, and it means that the next time something like this
happens, the machine will cleanly panic, and leave a very easy to
understand indication of what went wrong.
- Ted
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/
From: Andrea Arcangeli <and...@suse.de>
Subject: Re: timer_bh robusteness fix against potential deadlocks
Date: 2000/01/14
Message-ID: <fa.ikreegv.1bl0kr2@ifi.uio.no>#1/1
X-Deja-AN: 572766913
Original-Date: Fri, 14 Jan 2000 15:56:55 +0100 (CET)
Sender: owner-linux-ker...@vger.rutgers.edu
Original-Message-ID: <Pine.LNX.4.21.0001141552150.2653-100000@alpha.random>
References: <fa.hpj3vlv.752u12@ifi.uio.no>
X-PGP-Key-URL: http://e-mind.com/~andrea/aa.asc
To: ursus <ur...@usa.net>
X-Sender: and...@alpha.random
Content-Type: TEXT/PLAIN; charset=US-ASCII
X-Orcpt: rfc822;linux-kernel-outgoing-dig
Organization: Internet mailing list
MIME-Version: 1.0
X-GnuPG-Key-URL: http://e-mind.com/~andrea/aa.gnupg.asc
Newsgroups: fa.linux.kernel
X-Loop: majord...@vger.rutgers.edu
On Wed, 12 Jan 2000, Andrea Arcangeli wrote:
>+ while (timer) {
> void (*fn)(unsigned long) = timer->function;
> unsigned long data = timer->data;
>- detach_timer(timer);
>- timer->next = timer->prev = NULL;
>+ struct timer_list * tmp = timer;
>+ timer = timer->next;
>+ detach_timer(tmp);
>+ tmp->next = tmp->prev = NULL;
> spin_unlock_irq(&timerlist_lock);
If at this point the timer pointed by "timer" gets detached while "fn" is
running, at the next loop the machine is going to fail. I am sorry.
>Or you can download the patch from here:
>
> ftp://ftp.*.kernel.org/pub/linux/kernel/people/andrea/patches/v2.2/2.2.14/timer_bh-deadlock-1.gz
> ftp://ftp.*.kernel.org/pub/linux/kernel/people/andrea/patches/v2.3/2.3.38/timer_bh-deadlock-1.gz
Please don't use the two timer_bh patches quoted above (neither the -2
optimized version). Having the timer robust against buggy users is not
necessary but only desiderable, so actually you don't need it.
Nevertheless I'll fix the problem soon.
FYI: the delack-timer-3 patch here:
ftp://ftp.*.kernel.org/pub/linux/kernel/people/andrea/patches/v2.2/2.2.14/delack-timer-3.gz
seems to fix the wait_on_bh popular deadlock on UP/SMP webservers fine :).
Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/
From: ty...@valinux.com Subject: Re: timer_bh robusteness fix against potential deadlocks Date: 2000/01/15 Message-ID: <fa.i6sti0v.cjc93u@ifi.uio.no>#1/1 X-Deja-AN: 572800041 Original-Date: Tue, 11 Jan 2000 18:03:40 -0800 Sender: owner-linux-ker...@vger.rutgers.edu Original-Message-Id: <E128D8O-0007ai-00@dcl.su.varesearch.com> References: <fa.hs37vlv.2l2u1e@ifi.uio.no> To: and...@suse.de X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list Phone: (781) 391-3464 Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu Date: Wed, 12 Jan 2000 02:49:56 +0100 (CET) From: Andrea Arcangeli <and...@suse.de> As TyTso suggested the problem could be ato to be zero. I checked and I finally also spotted the offending bug that was causing the above condition to happen (second chunk of the patch): --- 2.2.14-tcp/net/ipv4/tcp_output.c.~1~ Fri Jan 7 18:19:25 2000 +++ 2.2.14-tcp/net/ipv4/tcp_output.c Wed Jan 12 02:47:32 2000 @@ -1004,7 +1004,7 @@ unsigned long timeout; /* Stay within the limit we were given */ - timeout = tp->ato; + timeout = (tp->ato << 1) >> 1; if (timeout > max_timeout) timeout = max_timeout; timeout += jiffies; Note that max_timeout is always a small positive number (HZ/2 or HZ/10), and timeout is a unsigned long. Hence if the quickack bit is set, timeout is > max_timeout, and so timeout gets capped to max_timeout. Without the patch, we simply delay the hack by the max_timeout instead of the current value of ato. Probably not the best, but not a disaster, either. @@ -1044,6 +1044,8 @@ */ if(tcp_in_quickack_mode(tp)) tcp_exit_quickack_mode(tp); + if (!tp->ato) + tp->ato = tp->rto; tcp_send_delayed_ack(tp, HZ/2); return; } This fixes the bug, but I'd be much happier if we put a belt-and-suspenders check in tcp_send_delayed_ack. If there's some other place which allows tcp_send_delayed_ack() to be called with tp->ato set to zero, we shouldn't lock up the entire kernel. So I'd propose adding to the tcp_send_delayed_ack() something like this: #define min_timeout HZ/50 if (timeout < min_timeout) timeout = min_timeout; /* This prevents an endless kernel loop */ (who me, paranoid?) - Ted - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
From: <arij...@valinux.com>
Subject: Re: timer_bh robusteness fix against potential deadlocks
Date: 2000/01/15
Message-ID: <fa.lcv2c3v.1r0obru@ifi.uio.no>#1/1
X-Deja-AN: 572878309
Original-Date: Fri, 14 Jan 2000 17:43:21 -0500 (EST)
Sender: owner-linux-ker...@vger.rutgers.edu
Original-Message-ID: <Pine.LNX.4.10.10001141737580.1356-100000@lap.valinuxny.net>
References: <fa.ikreegv.1bl0kr2@ifi.uio.no>
To: Andrea Arcangeli <and...@suse.de>
X-Sender: arij...@lap.valinuxny.net
X-Authentication-Warning: lap.valinuxny.net: arijort owned process doing -bs
Content-Type: TEXT/PLAIN; charset=US-ASCII
X-Orcpt: rfc822;linux-kernel-outgoing-dig
Organization: Internet mailing list
MIME-Version: 1.0
Newsgroups: fa.linux.kernel
X-Loop: majord...@vger.rutgers.edu
Andrea,
Just to be clear...
Are you saying that the entire timer_bh-deadlock patch is bad?
Or simply the hunk that you refer to below, which begins like this:
@@ -1135,17 +1165,20 @@
ftp://ftp.*.kernel.org/pub/linux/kernel/people/andrea/patches/v2.2/2.2.14/timer_bh-deadlock-1.gz
I'm assuming it's the whole patch.
ari
On Fri, 14 Jan 2000, Andrea Arcangeli wrote:
> On Wed, 12 Jan 2000, Andrea Arcangeli wrote:
>
> >+ while (timer) {
> > void (*fn)(unsigned long) = timer->function;
> > unsigned long data = timer->data;
> >- detach_timer(timer);
> >- timer->next = timer->prev = NULL;
> >+ struct timer_list * tmp = timer;
> >+ timer = timer->next;
> >+ detach_timer(tmp);
> >+ tmp->next = tmp->prev = NULL;
> > spin_unlock_irq(&timerlist_lock);
>
> If at this point the timer pointed by "timer" gets detached while "fn" is
> running, at the next loop the machine is going to fail. I am sorry.
>
> >Or you can download the patch from here:
> >
> > ftp://ftp.*.kernel.org/pub/linux/kernel/people/andrea/patches/v2.2/2.2.14/timer_bh-deadlock-1.gz
> > ftp://ftp.*.kernel.org/pub/linux/kernel/people/andrea/patches/v2.3/2.3.38/timer_bh-deadlock-1.gz
>
> Please don't use the two timer_bh patches quoted above (neither the -2
> optimized version). Having the timer robust against buggy users is not
> necessary but only desiderable, so actually you don't need it.
> Nevertheless I'll fix the problem soon.
>
> FYI: the delack-timer-3 patch here:
>
> ftp://ftp.*.kernel.org/pub/linux/kernel/people/andrea/patches/v2.2/2.2.14/delack-timer-3.gz
>
> seems to fix the wait_on_bh popular deadlock on UP/SMP webservers fine :).
>
> Andrea
>
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/
From: Andrea Arcangeli <and...@suse.de> Subject: Re: timer_bh robusteness fix against potential deadlocks Date: 2000/01/15 Message-ID: <fa.jmunc1v.bgikqb@ifi.uio.no>#1/1 X-Deja-AN: 572924322 Original-Date: Sat, 15 Jan 2000 01:46:12 +0100 (CET) Sender: owner-linux-ker...@vger.rutgers.edu Original-Message-ID: <Pine.LNX.4.21.0001150145061.14161-100000@alpha.random> References: <fa.lcv2c3v.1r0obru@ifi.uio.no> X-PGP-Key-URL: http://e-mind.com/~andrea/aa.asc To: arij...@valinux.com X-Sender: and...@alpha.random Content-Type: TEXT/PLAIN; charset=US-ASCII X-Orcpt: rfc822;linux-kernel-outgoing-dig Organization: Internet mailing list MIME-Version: 1.0 X-GnuPG-Key-URL: http://e-mind.com/~andrea/aa.gnupg.asc Newsgroups: fa.linux.kernel X-Loop: majord...@vger.rutgers.edu On Fri, 14 Jan 2000 arij...@valinux.com wrote: >Are you saying that the entire timer_bh-deadlock patch is bad? >Or simply the hunk that you refer to below, which begins like this: All. You can safely reverse such patch completly because it's not necessary for now. >ftp://ftp.*.kernel.org/pub/linux/kernel/people/andrea/patches/v2.2/2.2.14/timer_bh-deadlock-1.gz > >I'm assuming it's the whole patch. You are correct. The whole patch. Andrea - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/