Bug#478062: This has a work around
Ilpo Järvinen wrote:
Ah, there were some other bug reports as well to discover...
snip
The command you used was exactly what I was asking for (I don't know what
else you expected to work). :-)
So the problem was 100% reproducable for you with FRTO (I'm still catching
up the details here :-))?
Yes, 100% reproducible if you turn FRTO = 2. Printing works as it is
supposed to if FRTO = 0
snip
Yeah, you found out a right one. :-)
I suspect there's some corner case which might be buggy in kernel, but
there are other possibilities currently as well. I'd like to get this
fixed if it's a kernel bug, and think about work-arounds if it seems to be
elsewhere (end-point or middlebox). It might be that those devices just
blantantly discard any out-of-order segments, it could explain this kind
of phenomena, but we'll soon see what's doing
I also suspect a corner case. I suspect the hardware used has a "narrow
window" in which to allow the network traffic and the *new* kernels for
some reason are outside this window. However, just because FC8, Ubuntu
8.04 and Debian all see the same thing does not rule out an issue with
cupsys.
.
snip
Thanks a lot for the help. Yes, it's possible to get basically all
unrelated things filtered out, and I'm ok even with a log that has
IP-addresses anonymized (e.g., with sed afterwards) if you feel a need
to do so.
Turn FRTO back on (set the tcp_frto to 2). Then run this command (as
superuser):
tcpdump -i <interface> -w frtoprob.log host <printerip> and host <localinterfaceip>
...That will exclude everything else but stuff between the relevant hosts,
you will get a file called frtoprob.log. Please change those <>-marked
parts so that they match you setup. I suppose that printerip should be
192.168.200.150 for your case (based on debian bug). ...Then just
reproduce the problem.
Please wait at least minutes before ctrl-c:ing the tcpdump! If you really
want to capture the whole flow, you can keep the tcpdump running until the
connection between <localinterfaceip> and <printerip> has disappeared
from netstat's output (might take considerable amount of time, 20mins or
so if it slowly makes progress whole the time), though I think I can
determine the problem pattern with somewhat less than that, e.g., 5mins
during the problem should probably be enough.
You can evaluate the generated log afterwards with tcpdump -r frtoprob.log
(superuser rights are no longer mandatory, as long as the generated log
file is accessable for an ordinary user). I'd like a verbose log, use this
cmd:
tcpdump -tt -vvv -n -r frtoprob.log > frtoprob.txt
(You could add
| sed -e 's|<printerip>|prnt|g' -e 's|<localifaceip>|myip|g'
there before redirection if you would want to hide the ip addresses too,
but since you have one in debian bug too, I suspect you don't feel that
necessary).
Send what is in the frtoprob.txt, preferrable in such format that there
won't be extra newlines (yes, the lines are long) :-). The generated
output doesn't contain even any payload between the printer and your
machine, just protocol headers printed out (the binary dump would contain
part of the payload), not to speak of anything between any other hosts.
...I hope I wasn't overly verbose here. :-)
Just long enough :)
I suppose you're willing to test some kernel patch as well if that becomes
necessary?
Yup. I found it. I *own* it. This impacts a very small amount of
people as evidenced by the resounding silence of complaints about it
(which is shy I think the hardware has a "small window" to accept
packets) I will have to dust off my kernel patching skills, been a long
long time since I had to patch a kernel.
Interesting: I have stopped the cups print job, I have reset the
printer, powered off the printerserver device, and re-seated the network
cable (required to clear the stuck tcp packets so that the other packets
can get to the printer), I have stopped cupsysd and netstat still shows
an tcp connection with 192.168.200.150 (printer). I include this info
as this might be needed by you. I suspect that that is not *normal*
behavior, but I have never ran netstat --tcp -c on a print job that
failed, so I don't know:
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address
State
tcp 0 7168 dam-main.local:35770 192.168.200.150:9100
ESTABLISHED
Ran the tcpdump for a solid 20 minutes, attached.
Let me know if you need anything else.
Thanks!
--
Damon L. Chesser
damon@damtek.com
http://www.linkedin.com/in/dchesser
1210168073.605885 arp who-has 192.168.200.150 tell 192.168.200.15
1210168073.609015 arp reply 192.168.200.150 is-at 00:0e:3b:00:3a:75
1210168073.609061 IP (tos 0x0, ttl 64, id 63223, offset 0, flags [DF], proto TCP (6), length 60) 192.168.200.15.35770 > 192.168.200.150.9100: S, cksum 0xc0b9 (correct), 2773183952:2773183952(0) win 5840 <mss 1460,sackOK,timestamp 16096039 0,nop,wscale 7>
1210168073.611728 IP (tos 0x0, ttl 100, id 21, offset 0, flags [none], proto TCP (6), length 44) 192.168.200.150.9100 > 192.168.200.15.35770: S, cksum 0x94a7 (correct), 214441984:214441984(0) ack 2773183953 win 1024 <mss 1024>
1210168073.611810 IP (tos 0x0, ttl 64, id 63224, offset 0, flags [DF], proto TCP (6), length 40) 192.168.200.15.35770 > 192.168.200.150.9100: ., cksum 0x97e0 (correct), 1:1(0) ack 1 win 5840
1210168075.424729 IP (tos 0x0, ttl 64, id 63225, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: . 1:513(512) ack 1 win 5840
1210168075.424750 IP (tos 0x0, ttl 64, id 63226, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 513:1025(512) ack 1 win 5840
1210168075.433410 IP (tos 0x0, ttl 100, id 22, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0xa8b0 (correct), 1:1(0) ack 513 win 1024
1210168075.433465 IP (tos 0x0, ttl 64, id 63227, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: . 1025:1537(512) ack 1 win 5840
1210168075.637881 IP (tos 0x0, ttl 64, id 63228, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 513:1025(512) ack 1 win 5840
1210168075.643234 IP (tos 0x0, ttl 100, id 23, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0xa6b0 (correct), 1:1(0) ack 1025 win 1024
1210168075.643293 IP (tos 0x0, ttl 64, id 63229, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 1537:2049(512) ack 1 win 5840
1210168076.058858 IP (tos 0x0, ttl 64, id 63230, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: . 1025:1537(512) ack 1 win 5840
1210168076.065195 IP (tos 0x0, ttl 100, id 24, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0xa4b0 (correct), 1:1(0) ack 1537 win 1024
1210168076.065218 IP (tos 0x0, ttl 64, id 63231, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: . 2049:2561(512) ack 1 win 5840
1210168076.764849 IP (tos 0x0, ttl 100, id 25, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0xa4b0 (correct), 1:1(0) ack 1537 win 1024
1210168076.894877 IP (tos 0x0, ttl 64, id 63232, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 1537:2049(512) ack 1 win 5840
1210168076.900138 IP (tos 0x0, ttl 100, id 26, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0xa2b0 (correct), 1:1(0) ack 2049 win 1024
1210168076.900171 IP (tos 0x0, ttl 64, id 63233, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 2561:3073(512) ack 1 win 5840
1210168077.600238 IP (tos 0x0, ttl 100, id 27, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0xa2b0 (correct), 1:1(0) ack 2049 win 1024
1210168078.562871 IP (tos 0x0, ttl 64, id 63234, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: . 2049:2561(512) ack 1 win 5840
1210168078.568982 IP (tos 0x0, ttl 100, id 28, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0xa0b0 (correct), 1:1(0) ack 2561 win 1024
1210168078.569027 IP (tos 0x0, ttl 64, id 63235, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: . 3073:3585(512) ack 1 win 5840
1210168079.263714 IP (tos 0x0, ttl 100, id 29, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0xa0b0 (correct), 1:1(0) ack 2561 win 1024
1210168081.893871 IP (tos 0x0, ttl 64, id 63236, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 2561:3073(512) ack 1 win 5840
1210168081.897744 IP (tos 0x0, ttl 100, id 30, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x9eb0 (correct), 1:1(0) ack 3073 win 1024
1210168081.897775 IP (tos 0x0, ttl 64, id 63237, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 3585:4097(512) ack 1 win 5840
1210168082.597179 IP (tos 0x0, ttl 100, id 31, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x9eb0 (correct), 1:1(0) ack 3073 win 1024
1210168088.549863 IP (tos 0x0, ttl 64, id 63238, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: . 3073:3585(512) ack 1 win 5840
1210168088.555052 IP (tos 0x0, ttl 100, id 32, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x9cb0 (correct), 1:1(0) ack 3585 win 1024
1210168088.555071 IP (tos 0x0, ttl 64, id 63239, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: . 4097:4609(512) ack 1 win 5840
1210168089.254224 IP (tos 0x0, ttl 100, id 33, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x9cb0 (correct), 1:1(0) ack 3585 win 1024
1210168101.865871 IP (tos 0x0, ttl 64, id 63240, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 3585:4097(512) ack 1 win 5840
1210168101.872129 IP (tos 0x0, ttl 100, id 34, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x9ab0 (correct), 1:1(0) ack 4097 win 1024
1210168101.872159 IP (tos 0x0, ttl 64, id 63241, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 4609:5121(512) ack 1 win 5840
1210168102.568296 IP (tos 0x0, ttl 100, id 35, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x9ab0 (correct), 1:1(0) ack 4097 win 1024
1210168128.494871 IP (tos 0x0, ttl 64, id 63242, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: . 4097:4609(512) ack 1 win 5840
1210168128.501748 IP (tos 0x0, ttl 100, id 36, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x98b0 (correct), 1:1(0) ack 4609 win 1024
1210168128.501779 IP (tos 0x0, ttl 64, id 63243, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: . 5121:5633(512) ack 1 win 5840
1210168129.196598 IP (tos 0x0, ttl 100, id 37, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x98b0 (correct), 1:1(0) ack 4609 win 1024
1210168181.746872 IP (tos 0x0, ttl 64, id 63244, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 4609:5121(512) ack 1 win 5840
1210168181.753119 IP (tos 0x0, ttl 100, id 41, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x96b0 (correct), 1:1(0) ack 5121 win 1024
1210168181.753151 IP (tos 0x0, ttl 64, id 63245, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 5633:6145(512) ack 1 win 5840
1210168182.452827 IP (tos 0x0, ttl 100, id 42, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x96b0 (correct), 1:1(0) ack 5121 win 1024
1210168288.246871 IP (tos 0x0, ttl 64, id 63246, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: . 5121:5633(512) ack 1 win 5840
1210168288.251178 IP (tos 0x0, ttl 100, id 46, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x94b0 (correct), 1:1(0) ack 5633 win 1024
1210168288.251213 IP (tos 0x0, ttl 64, id 63247, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: . 6145:6657(512) ack 1 win 5840
1210168288.950675 IP (tos 0x0, ttl 100, id 47, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x94b0 (correct), 1:1(0) ack 5633 win 1024
1210168408.249876 IP (tos 0x0, ttl 64, id 63248, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 5633:6145(512) ack 1 win 5840
1210168408.255523 IP (tos 0x0, ttl 100, id 54, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x92b0 (correct), 1:1(0) ack 6145 win 1024
1210168408.255556 IP (tos 0x0, ttl 64, id 63249, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 6657:7169(512) ack 1 win 5840
1210168408.950279 IP (tos 0x0, ttl 100, id 55, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x92b0 (correct), 1:1(0) ack 6145 win 1024
1210168528.253909 IP (tos 0x0, ttl 64, id 63250, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: . 6145:6657(512) ack 1 win 5840
1210168528.260088 IP (tos 0x0, ttl 100, id 59, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x90b0 (correct), 1:1(0) ack 6657 win 1024
1210168528.260126 IP (tos 0x0, ttl 64, id 63251, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 7169:7681(512) ack 1 win 5840
1210168528.954815 IP (tos 0x0, ttl 100, id 60, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x90b0 (correct), 1:1(0) ack 6657 win 1024
1210168648.257875 IP (tos 0x0, ttl 64, id 63252, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 6657:7169(512) ack 1 win 5840
1210168648.264753 IP (tos 0x0, ttl 100, id 67, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x8eb0 (correct), 1:1(0) ack 7169 win 1024
1210168648.264804 IP (tos 0x0, ttl 64, id 63253, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 7681:8193(512) ack 1 win 5840
1210168648.959350 IP (tos 0x0, ttl 100, id 68, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x8eb0 (correct), 1:1(0) ack 7169 win 1024
1210168768.261878 IP (tos 0x0, ttl 64, id 63254, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 7169:7681(512) ack 1 win 5840
1210168768.265497 IP (tos 0x0, ttl 100, id 72, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x8cb0 (correct), 1:1(0) ack 7681 win 1024
1210168768.265528 IP (tos 0x0, ttl 64, id 63255, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: . 8193:8705(512) ack 1 win 5840
1210168768.964044 IP (tos 0x0, ttl 100, id 73, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x8cb0 (correct), 1:1(0) ack 7681 win 1024
1210168888.262884 IP (tos 0x0, ttl 64, id 63256, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 7681:8193(512) ack 1 win 5840
1210168888.268781 IP (tos 0x0, ttl 100, id 77, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x8ab0 (correct), 1:1(0) ack 8193 win 1024
1210168888.268847 IP (tos 0x0, ttl 64, id 63257, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 8705:9217(512) ack 1 win 5840
1210168888.963510 IP (tos 0x0, ttl 100, id 78, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x8ab0 (correct), 1:1(0) ack 8193 win 1024
1210169008.265876 IP (tos 0x0, ttl 64, id 63258, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: . 8193:8705(512) ack 1 win 5840
1210169008.270775 IP (tos 0x0, ttl 100, id 85, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x88b0 (correct), 1:1(0) ack 8705 win 1024
1210169008.270808 IP (tos 0x0, ttl 64, id 63259, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: . 9217:9729(512) ack 1 win 5840
1210169008.968045 IP (tos 0x0, ttl 100, id 86, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x88b0 (correct), 1:1(0) ack 8705 win 1024
1210169128.269900 IP (tos 0x0, ttl 64, id 63260, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 8705:9217(512) ack 1 win 5840
1210169128.273712 IP (tos 0x0, ttl 100, id 90, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x86b0 (correct), 1:1(0) ack 9217 win 1024
1210169128.273744 IP (tos 0x0, ttl 64, id 63261, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 9729:10241(512) ack 1 win 5840
1210169128.972707 IP (tos 0x0, ttl 100, id 91, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x86b0 (correct), 1:1(0) ack 9217 win 1024
1210169248.269882 IP (tos 0x0, ttl 64, id 63262, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: . 9217:9729(512) ack 1 win 5840
1210169248.273988 IP (tos 0x0, ttl 100, id 98, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x84b0 (correct), 1:1(0) ack 9729 win 1024
1210169248.274020 IP (tos 0x0, ttl 64, id 63263, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: . 10241:10753(512) ack 1 win 5840
1210169248.972183 IP (tos 0x0, ttl 100, id 99, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x84b0 (correct), 1:1(0) ack 9729 win 1024
1210169368.274878 IP (tos 0x0, ttl 64, id 63264, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 9729:10241(512) ack 1 win 5840
1210169368.282107 IP (tos 0x0, ttl 100, id 103, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x82b0 (correct), 1:1(0) ack 10241 win 1024
1210169368.282164 IP (tos 0x0, ttl 64, id 63265, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.35770 > 192.168.200.150.9100: P 10753:11265(512) ack 1 win 5840
1210169368.976736 IP (tos 0x0, ttl 100, id 104, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.35770: ., cksum 0x82b0 (correct), 1:1(0) ack 10241 win 1024
Reply to: