Bug#684364: target: reports vfs_writev() returned -28 on iscsi activity
Package: linux-image-3.2.0-3-amd64
Version: 3.2.21-3
Severity: normal
Hello,
i'm using this virtual machine to export iscsi targets for windows machines to backup.
It has 620GB btrfs partition with 550GB iscsi file_io LUN exported.
I'm using btrfs snapshots to transport (consistent copy of) this big file to secondary backup storage (qnap and usb disk) - it takes around 12 hours to transport - thats reason to use snapshots.
Everything was working fine, until i run out of space on btrfs (because of snapshot and fullbackup from windows combination).
Since than, i have created fresh filesystem and create empty new file to export over iscsi.
Now, around 250GB of writen data do iscsi file, kern.log starts to fill with these messages
vfs_writev() returned -28
It keeps to show on every access to iscsi (even "rescan disk" operation in windows triggers it)
I was suspecting btrfs, but there is no oops in kern.log.
I tried to google this error and found nothing relevant.
I tried to reset everything (new filesystem, empty file, remove/add target from windows) and this errors comes back after writing approx. 250GB of data.
(Before problem, file was filled to around 540GB).
Size now:
# du -cms iscsiSBS
244561 iscsiSBS
Output of
'find /lib/modules/ | xargs strings | grep "vfs_writev() returned"'
shows only one line, which is in target_core_file.ko
With regards
Libor
-- Package-specific info:
** Version:
Linux version 3.2.0-3-amd64 (Debian 3.2.21-3) (debian-kernel@lists.debian.org) (gcc version 4.6.3 (Debian 4.6.3-8) ) #1 SMP Thu Jun 28 09:07:26 UTC 2012
** Command line:
BOOT_IMAGE=/boot/vmlinuz-3.2.0-3-amd64 root=/dev/mapper/vgDisk1-lvSystem ro quiet
** Tainted: O (4096)
* Out-of-tree module has been loaded.
** Kernel log:
[77575.095989] vfs_writev() returned -28
[77575.096650] vfs_writev() returned -28
[77575.097270] vfs_writev() returned -28
[77576.594098] vfs_writev() returned -28
[77580.197458] vfs_writev() returned -28
[77580.198214] vfs_writev() returned -28
[77585.345379] vfs_writev() returned -28
[77585.346186] vfs_writev() returned -28
[77585.346892] vfs_writev() returned -28
[77585.347642] vfs_writev() returned -28
** Model information
sys_vendor: VMware, Inc.
product_name: VMware Virtual Platform
product_version: None
chassis_vendor: No Enclosure
chassis_version: N/A
bios_vendor: Phoenix Technologies LTD
bios_version: 6.00
board_vendor: Intel Corporation
board_name: 440BX Desktop Reference Platform
board_version: None
** Loaded modules:
des_generic
ecb
md4
hmac
nls_utf8
cifs
autofs4
nfnetlink_log
nfnetlink
tcm_loop
tcm_fc
iscsi_target_mod
target_core_pscsi
target_core_file
target_core_iblock
target_core_mod
libfc
scsi_transport_fc
scsi_tgt
configfs
vmsync(O)
vmhgfs(O)
nfsd
nfs
nfs_acl
auth_rpcgss
fscache
lockd
sunrpc
dm_round_robin
dm_multipath
scsi_dh
loop
joydev
usbhid
hid
ses
enclosure
coretemp
crc32c_intel
ghash_clmulni_intel
aesni_intel
aes_x86_64
aes_generic
psmouse
snd_pcm
snd_page_alloc
snd_timer
snd
parport_pc
parport
soundcore
vmw_balloon
evdev
cryptd
i2c_piix4
ac
serio_raw
pcspkr
i2c_core
shpchp
power_supply
vmci(O)
processor
container
thermal_sys
button
ext4
crc16
jbd2
mbcache
btrfs
crc32c
libcrc32c
zlib_deflate
usb_storage
uas
dm_mod
sd_mod
crc_t10dif
uhci_hcd
sr_mod
cdrom
mptspi
scsi_transport_spi
mptscsih
ata_generic
floppy
mptbase
ehci_hcd
usbcore
ata_piix
usb_common
e1000
libata
scsi_mod
** Network interface configuration:
auto lo
iface lo inet loopback
allow-hotplug eth0
iface eth0 inet static
address 192.168.5.213
netmask 255.255.255.0
network 192.168.5.0
broadcast 192.168.5.255
** Network status:
*** IP interfaces and addresses:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:59:fe:c9 brd ff:ff:ff:ff:ff:ff
inet 192.168.5.213/24 brd 192.168.5.255 scope global eth0
inet6 fe80::20c:29ff:fe59:fec9/64 scope link
valid_lft forever preferred_lft forever
*** Device statistics:
Inter-| Receive | Transmit
face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed
lo: 87464948 10977 0 0 0 0 0 0 87464948 10977 0 0 0 0 0 0
eth0: 287037645968 197285259 0 24 0 0 0 0 8008550193 22837748 0 0 0 0 0 0
*** Protocol statistics:
Ip:
13169750 total packets received
0 forwarded
0 incoming packets discarded
13165208 incoming packets delivered
21135011 requests sent out
5992 reassemblies required
1498 packets reassembled ok
Icmp:
1803 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
destination unreachable: 10
echo requests: 1793
1793 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
echo replies: 1793
IcmpMsg:
InType3: 10
InType8: 1793
OutType0: 1793
Tcp:
162 active connections openings
3704 passive connection openings
0 failed connection attempts
19 connection resets received
5 connections established
12968199 segments received
22838104 segments send out
5078 segments retransmited
0 bad segments received.
9069 resets sent
Udp:
1070 packets received
0 packets to unknown port received.
0 packet receive errors
1070 packets sent
UdpLite:
TcpExt:
3895 invalid SYN cookies received
9 ICMP packets dropped because they were out-of-window
82 TCP sockets finished time wait in fast timer
6439 delayed acks sent
79 delayed acks further delayed because of locked socket
Quick ack mode was activated 110 times
10069295 packets directly queued to recvmsg prequeue.
5939254 bytes directly in process context from backlog
344335434 bytes directly received in process context from prequeue
4716149 packet headers predicted
1544725 packets header predicted and directly queued to user
1437481 acknowledgments not containing data payload received
3743691 predicted acknowledgments
4321 times recovered from packet loss by selective acknowledgements
2 bad SACK blocks received
Detected reordering 1 times using FACK
Detected reordering 51 times using SACK
Detected reordering 2 times using time stamp
966 congestion windows fully recovered without slow start
14 congestion windows partially recovered using Hoe heuristic
3365 congestion windows recovered without slow start by DSACK
118 congestion windows recovered without slow start after partial ack
5 TCP data loss events
1 timeouts after SACK recovery
4490 fast retransmits
216 forward retransmits
38 retransmits in slow start
251 other TCP timeouts
1 times receiver scheduled too late for direct processing
300 DSACKs sent for old packets
4812 DSACKs received
3639 connections reset due to unexpected data
17 connections reset due to early user close
1 connections aborted due to timeout
TCPDSACKIgnoredNoUndo: 13
TCPSackShifted: 18491
TCPSackMerged: 10435
TCPSackShiftFallback: 52223
IpExt:
InMcastPkts: 9
InBcastPkts: 198621
InOctets: 2116677939
OutOctets: -883354183
InMcastOctets: 324
InBcastOctets: 23668880
** PCI devices:
Virtual HW in vmware esxi 5.0, no pvscsi nor vmxnet
** USB devices:
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 003: ID 1058:070c Western Digital Technologies, Inc.
Bus 002 Device 002: ID 0e0f:0003 VMware, Inc. Virtual Mouse
Bus 002 Device 003: ID 0e0f:0002 VMware, Inc. Virtual USB Hub
-- System Information:
Debian Release: wheezy/sid
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Kernel: Linux 3.2.0-3-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages linux-image-3.2.0-3-amd64 depends on:
ii debconf [debconf-2.0] 1.5.44
ii initramfs-tools [linux-initramfs-tool] 0.107
ii kmod 8-2
ii linux-base 3.5
ii module-init-tools 8-2
Versions of packages linux-image-3.2.0-3-amd64 recommends:
pn firmware-linux-free <none>
Versions of packages linux-image-3.2.0-3-amd64 suggests:
pn debian-kernel-handbook <none>
ii grub-pc 1.99-22.1
pn linux-doc-3.2 <none>
Versions of packages linux-image-3.2.0-3-amd64 is related to:
pn firmware-atheros <none>
pn firmware-bnx2 <none>
pn firmware-bnx2x <none>
pn firmware-brcm80211 <none>
pn firmware-intelwimax <none>
pn firmware-ipw2x00 <none>
pn firmware-ivtv <none>
pn firmware-iwlwifi <none>
pn firmware-libertas <none>
pn firmware-linux <none>
pn firmware-linux-nonfree <none>
pn firmware-myricom <none>
pn firmware-netxen <none>
pn firmware-qlogic <none>
pn firmware-ralink <none>
pn firmware-realtek <none>
pn xen-hypervisor <none>
-- debconf information:
linux-image-3.2.0-3-amd64/postinst/depmod-error-initrd-3.2.0-3-amd64: false
linux-image-3.2.0-3-amd64/postinst/ignoring-ramdisk:
linux-image-3.2.0-3-amd64/postinst/missing-firmware-3.2.0-3-amd64:
linux-image-3.2.0-3-amd64/prerm/removing-running-kernel-3.2.0-3-amd64: true
Reply to: