[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: GlusterFS

Разобрался сам:

Во-первых, надо перед подключением каждой новой ноды делать сперва

gluster peer probe <node IP\name>

А после монтировать вот так:

mount.glusterfs <node IP\name>:<volume_name> /mount/path
А вообще все маны про GlusterFS в сети устарели, копаюсь как некрофил какой.. В Дебиане уже всё гораздо проще сделано и управляется в 3 команды. 


07.12.2012 14:29, evgeny_veresk@mail.ru пишет:

Добрый день!
Кто-нибудь ставил это чудо кластерное? Запустил сервера на 2-х машинах, реальных, в сети. Каждая говорит так: 

# gluster volume info all

Volume Name: gf
Type: Distribute
Status: Started
Number of Bricks: 2
Transport-type: tcp,rdma
Bricks:
Brick1: 10.0.0.25:/srv/export
Brick2: 10.0.0.22:/srv/export

Всё красиво. Друг друга сервера тоже видят:

# gluster peer status
Number of Peers: 1

Hostname: 10.0.0.22
Uuid: 53d1b617-d83f-4374-9a3d-0676539e5482
State: Peer in Cluster (Connected)
Всё, типа, круто должно быть. А вот при попытке монтировать всё это дело зависает. Причём, пока не указал в конфиге порт подключения - ругалось на то, что порта не знает и знать не хочет. А ссылки только на багтрекер glusterfs ведут :-( 

Конфиг такой:

# cat /etc/glusterfs/glusterfs.vol
volume gf1
type protocol/client
option transport-type tcp
option remote-host 10.0.0.22
option remote-subvolume gf
option remote-port 24009
end-volume

volume gf2
type protocol/client
option transport-type tcp
option remote-host 10.0.0.25
option remote-subvolume gf
option remote-port 24009
end-volume

volume replicate
  type cluster/replicate
  subvolumes gf1 gf2
end-volume

volume writebehind
  type performance/write-behind
  option window-size 1MB
  subvolumes replicate
end-volume

volume cache
  type performance/io-cache
  option cache-size 512MB
  subvolumes writebehind
end-volume

А вот лог такой:

glusterfs -f /etc/glusterfs/glusterfs.vol /srv/gfs --debug
[2012-12-07 13:44:44.412053] I [glusterfsd.c:1493:main] 0-glusterfs: Started running glusterfs version 3.2.7 [2012-12-07 13:44:44.412488] D [xlator.c:1302:xlator_dynload] 0-xlator: dlsym(reconfigure) on /usr/lib/glusterfs/3.2.7/xlator/mount/fuse.so: undefined symbol: reconfigure -- neglecting [2012-12-07 13:44:44.412547] D [xlator.c:1308:xlator_dynload] 0-xlator: dlsym(validate_options) on /usr/lib/glusterfs/3.2.7/xlator/mount/fuse.so: undefined symbol: validate_options -- neglecting [2012-12-07 13:44:44.412593] D [glusterfsd.c:325:create_fuse_mount] 0-: fuse direct io type 2 [2012-12-07 13:44:44.414224] D [glusterfsd.c:374:get_volfp] 0-glusterfsd: loading volume file /etc/glusterfs/glusterfs.vol [2012-12-07 13:44:44.820940] D [xlator.c:1285:xlator_dynload] 0-xlator: dlsym(notify) on /usr/lib/glusterfs/3.2.7/xlator/performance/write-behind.so: undefined symbol: notify -- neglecting [2012-12-07 13:44:44.821117] D [xlator.c:1285:xlator_dynload] 0-xlator: dlsym(notify) on /usr/lib/glusterfs/3.2.7/xlator/performance/io-cache.so: undefined symbol: notify -- neglecting [2012-12-07 13:44:44.821235] W [xlator.c:1137:validate_xlator_volume_options] 0-writebehind: option 'window-size' is deprecated, preferred is 'cache-size', continuing with correction [2012-12-07 13:44:44.821285] W [io-cache.c:1720:init] 0-cache: dangling volume. check volfile [2012-12-07 13:44:44.821318] D [xlator.c:1949:xlator_get_volopt_info] 0-: Returning 0 [2012-12-07 13:44:44.821346] D [xlator.c:1949:xlator_get_volopt_info] 0-: Returning 0 [2012-12-07 13:44:44.821373] D [xlator.c:1949:xlator_get_volopt_info] 0-: Returning 0 [2012-12-07 13:44:44.821398] D [xlator.c:1949:xlator_get_volopt_info] 0-: Returning 0 [2012-12-07 13:44:44.821552] W [write-behind.c:3023:init] 0-writebehind: disabling write-behind for first 0 bytes [2012-12-07 13:44:44.821582] D [xlator.c:1949:xlator_get_volopt_info] 0-: Returning 0 [2012-12-07 13:44:44.821609] D [xlator.c:1949:xlator_get_volopt_info] 0-: Returning 0 [2012-12-07 13:44:44.824664] D [xlator.c:1949:xlator_get_volopt_info] 0-: Returning 0 [2012-12-07 13:44:44.824704] D [xlator.c:1949:xlator_get_volopt_info] 0-: Returning 0 [2012-12-07 13:44:44.824769] D [xlator.c:1949:xlator_get_volopt_info] 0-: Returning 0 [2012-12-07 13:44:44.824805] I [client.c:1989:build_client_config] 0-gf2: remote-port is 24009 [2012-12-07 13:44:44.824834] D [xlator.c:1949:xlator_get_volopt_info] 0-: Returning 0 [2012-12-07 13:44:44.826538] D [rpc-clnt.c:919:rpc_clnt_connection_init] 0-gf2: defaulting frame-timeout to 30mins [2012-12-07 13:44:44.826618] D [rpc-transport.c:673:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/glusterfs/3.2.7/rpc-transport/socket.so [2012-12-07 13:44:44.826870] D [rpc-transport.c:97:__volume_option_value_validate] 0-gf2: no range check required for 'option remote-port 24009' [2012-12-07 13:44:44.826940] D [rpc-clnt.c:1301:rpcclnt_cbk_program_register] 0-gf2: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 [2012-12-07 13:44:44.826972] D [client.c:2117:client_init_rpc] 0-gf2: client init successful [2012-12-07 13:44:44.827008] D [xlator.c:1949:xlator_get_volopt_info] 0-: Returning 0 [2012-12-07 13:44:44.827039] I [client.c:1989:build_client_config] 0-gf1: remote-port is 24009 [2012-12-07 13:44:44.827068] D [xlator.c:1949:xlator_get_volopt_info] 0-: Returning 0 [2012-12-07 13:44:44.829000] D [rpc-clnt.c:919:rpc_clnt_connection_init] 0-gf1: defaulting frame-timeout to 30mins [2012-12-07 13:44:44.829080] D [rpc-transport.c:673:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib/glusterfs/3.2.7/rpc-transport/socket.so [2012-12-07 13:44:44.829134] D [rpc-transport.c:97:__volume_option_value_validate] 0-gf1: no range check required for 'option remote-port 24009' [2012-12-07 13:44:44.829180] D [rpc-clnt.c:1301:rpcclnt_cbk_program_register] 0-gf1: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1 [2012-12-07 13:44:44.829210] D [client.c:2117:client_init_rpc] 0-gf1: client init successful [2012-12-07 13:44:44.829244] W [graph.c:120:_log_if_option_is_invalid] 0-gf2: option 'remote-port' is not recognized [2012-12-07 13:44:44.829277] W [graph.c:120:_log_if_option_is_invalid] 0-gf1: option 'remote-port' is not recognized [2012-12-07 13:44:44.829309] D [fuse-bridge.c:3479:notify] 0-fuse: got event 12 on graph 0 [2012-12-07 13:44:44.829348] I [client.c:1935:notify] 0-gf1: parent translators are ready, attempting connect on transport [2012-12-07 13:44:44.829392] D [name.c:157:client_fill_address_family] 0-gf1: address-family not specified, guessing it to be inet/inet6 [2012-12-07 13:44:44.830325] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.0.0.22 (port-24009) for hostname: 10.0.0.22 and port: 24009 [2012-12-07 13:44:44.830492] I [client.c:1935:notify] 0-gf2: parent translators are ready, attempting connect on transport [2012-12-07 13:44:44.830531] D [name.c:157:client_fill_address_family] 0-gf2: address-family not specified, guessing it to be inet/inet6 [2012-12-07 13:44:44.830950] D [common-utils.c:151:gf_resolve_ip6] 0-resolver: returning ip-10.0.0.25 (port-24009) for hostname: 10.0.0.25 and port: 24009 
Given volfile:
+------------------------------------------------------------------------------+ 
  1: volume gf1
  2: type protocol/client
  3: option transport-type tcp
  4: option remote-host 10.0.0.22
  5: option remote-subvolume gf
  6: option remote-port 24009
  7: end-volume
  8:
  9: volume gf2
 10: type protocol/client
 11: option transport-type tcp
 12: option remote-host 10.0.0.25
 13: option remote-subvolume gf
 14: option remote-port 24009
 15: end-volume
 16:
 17: volume replicate
 18:   type cluster/replicate
 19:   subvolumes gf1 gf2
 20: end-volume
 21:
 22: volume writebehind
 23:   type performance/write-behind
 24:   option window-size 1MB
 25:   subvolumes replicate
 26: end-volume
 27:
 28: volume cache
 29:   type performance/io-cache
 30:   option cache-size 512MB
 31:   subvolumes writebehind
 32: end-volume
+------------------------------------------------------------------------------+ [2012-12-07 13:44:44.831476] D [client.c:1856:client_rpc_notify] 0-gf1: got RPC_CLNT_CONNECT [2012-12-07 13:44:44.831580] D [client-handshake.c:179:client_start_ping] 0-gf1: returning as transport is already disconnected OR there are no frames (1 || 1) [2012-12-07 13:44:44.831639] D [client.c:1856:client_rpc_notify] 0-gf2: got RPC_CLNT_CONNECT [2012-12-07 13:44:44.831798] D [client-handshake.c:179:client_start_ping] 0-gf2: returning as transport is already disconnected OR there are no frames (1 || 1) [2012-12-07 13:44:44.831884] I [client-handshake.c:1090:select_server_supported_programs] 0-gf1: Using Program GlusterFS 3.2.7, Num (1298437), Version (310) [2012-12-07 13:44:44.831966] D [client-handshake.c:179:client_start_ping] 0-gf1: returning as transport is already disconnected OR there are no frames (1 || 1) [2012-12-07 13:44:44.832022] I [client-handshake.c:1090:select_server_supported_programs] 0-gf2: Using Program GlusterFS 3.2.7, Num (1298437), Version (310) [2012-12-07 13:44:44.832198] D [client-handshake.c:179:client_start_ping] 0-gf2: returning as transport is already disconnected OR there are no frames (1 || 1) [2012-12-07 13:44:44.832263] W [client-handshake.c:836:client_setvolume_cbk] 0-gf2: failed to set the volume (No such file or directory) [2012-12-07 13:44:44.832300] W [client-handshake.c:862:client_setvolume_cbk] 0-gf2: failed to get 'process-uuid' from reply dict [2012-12-07 13:44:44.832328] E [client-handshake.c:868:client_setvolume_cbk] 0-gf2: SETVOLUME on remote-host failed: remote-subvolume "gf" is not found [2012-12-07 13:44:44.832358] I [client-handshake.c:933:client_setvolume_cbk] 0-gf2: sending CHILD_CONNECTING event [2012-12-07 13:44:44.832405] W [client-handshake.c:836:client_setvolume_cbk] 0-gf1: failed to set the volume (No such file or directory) [2012-12-07 13:44:44.832440] W [client-handshake.c:862:client_setvolume_cbk] 0-gf1: failed to get 'process-uuid' from reply dict [2012-12-07 13:44:44.832468] E [client-handshake.c:868:client_setvolume_cbk] 0-gf1: SETVOLUME on remote-host failed: remote-subvolume "gf" is not found [2012-12-07 13:44:44.832497] I [client-handshake.c:933:client_setvolume_cbk] 0-gf1: sending CHILD_CONNECTING event [2012-12-07 13:44:44.832526] D [fuse-bridge.c:3479:notify] 0-fuse: got event 7 on graph 0 [2012-12-07 13:44:44.836127] I [fuse-bridge.c:3461:fuse_graph_setup] 0-fuse: switched to graph 0 [2012-12-07 13:44:44.836304] I [fuse-bridge.c:3049:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.17
^C[2012-12-07 13:45:27.231703] W [glusterfsd.c:727:cleanup_and_exit] (-->/lib/i386-linux-gnu/i686/cmov/libc.so.6(epoll_wait+0x56) [0xb75b11f6] (-->[0xb770c424] (-->[0xb770c400]))) 0-: received signum (2), shutting down [2012-12-07 13:45:27.231747] D [glusterfsd-mgmt.c:1076:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper signout arguments not given [2012-12-07 13:45:27.231772] I [fuse-bridge.c:3849:fini] 0-fuse: Unmounting '/srv/gfs'.
Reply to: