Server Admin Авторский блог системного администратора
доброго времени суток возникла у меня проблема после того как вырубили свет от кластера Proxmox отвалился веб интерфейс и чтобы я с ним не делал не мог его запустить, запустил виртуалки через консоль но это пол беды сегодня просто напросто на одной из нод кластера на которой крутился контролер домена просто сначала повисла а потом и отказалась запускаться виртуалка
при наборе qm list машина задумывается очень и очень на долго либо открывать ещо одну консоль либо если это напрямую сделано то рестартовать
qm start (id) так же ни к чему не приводит ради интереса ждал пол дня и ничего
так же ни одна из трёх нод в кластере не работает веб интерфейс я убрал кворум со всех нод но это ни к чему не привело просто перестал сыпать ошибку что нет кворума
я уже честно не знаю в какую сторону копать
подскажите кто чем может
@srubtsov надо логи смотреть, что там за ошибки и почему веб интерфейс не открывается. Но раз и виртуалки начали виснуть, там явно серверу поплохело от аварийного выключения. Без чтения логов не разобраться. Начать надо с системных - /var/log/syslog.
И самое главное - проверьте диск на ошибки!!! Хотя-бы встроенными инструментами.
вот скидываю syslog диск проверял никаких ошибок он не нашол
ec 29 22:26:26 pve212 corosync[961]: [TOTEM ] Token has not been received in 10 ms
Dec 29 22:27:06 pve212 smartd[661]: Device: /dev/sdb [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 64 to 65
Dec 29 22:27:06 pve212 smartd[661]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 36 to 35
Dec 29 22:40:35 pve212 corosync[961]: [TOTEM ] Token has not been received in 10 ms
Dec 29 22:40:35 pve212 corosync[961]: [TOTEM ] A new membership (1.43a0d) was formed. Members
Dec 29 22:40:35 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 22:40:35 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 22:40:35 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 22:40:35 pve212 corosync[961]: [QUORUM] Members[3]: 1 2 3
Dec 29 22:40:35 pve212 corosync[961]: [MAIN ] Completed service synchronization, ready to provide service.
Dec 29 22:45:39 pve212 corosync[961]: [KNET ] link: host: 2 link: 0 is down
Dec 29 22:45:39 pve212 corosync[961]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Dec 29 22:45:39 pve212 corosync[961]: [KNET ] host: host: 2 has no active links
Dec 29 22:45:40 pve212 corosync[961]: [TOTEM ] Token has not been received in 50 ms
Dec 29 22:45:41 pve212 corosync[961]: [KNET ] rx: host: 2 link: 0 is up
Dec 29 22:45:41 pve212 corosync[961]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Dec 29 22:45:41 pve212 corosync[961]: [TOTEM ] A new membership (1.43a11) was formed. Members
Dec 29 22:45:41 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 22:45:41 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 22:45:41 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 22:45:41 pve212 corosync[961]: [QUORUM] Members[3]: 1 2 3
Dec 29 22:45:41 pve212 corosync[961]: [MAIN ] Completed service synchronization, ready to provide service.
Dec 29 22:46:40 pve212 corosync[961]: [TOTEM ] Token has not been received in 27 ms
Dec 29 22:46:41 pve212 corosync[961]: [TOTEM ] A processor failed, forming new configuration.
Dec 29 22:46:41 pve212 corosync[961]: [TOTEM ] A new membership (1.43a15) was formed. Members
Dec 29 22:46:41 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 22:46:41 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 22:46:41 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 22:46:41 pve212 corosync[961]: [QUORUM] Members[3]: 1 2 3
Dec 29 22:46:41 pve212 corosync[961]: [MAIN ] Completed service synchronization, ready to provide service.
Dec 29 22:57:18 pve212 corosync[961]: [TOTEM ] Token has not been received in 24 ms
Dec 29 22:57:18 pve212 corosync[961]: [TOTEM ] A processor failed, forming new configuration.
Dec 29 22:57:19 pve212 corosync[961]: [TOTEM ] A new membership (1.43a19) was formed. Members
Dec 29 22:57:19 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 22:57:19 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 22:57:19 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 22:57:19 pve212 corosync[961]: [QUORUM] Members[3]: 1 2 3
Dec 29 22:57:19 pve212 corosync[961]: [MAIN ] Completed service synchronization, ready to provide service.
Dec 29 22:58:07 pve212 corosync[961]: [TOTEM ] Token has not been received in 30 ms
Dec 29 22:58:08 pve212 corosync[961]: [TOTEM ] A processor failed, forming new configuration.
Dec 29 22:58:08 pve212 corosync[961]: [TOTEM ] A new membership (1.43a1d) was formed. Members
Dec 29 22:58:08 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 22:58:08 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 22:58:08 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 22:58:08 pve212 corosync[961]: [QUORUM] Members[3]: 1 2 3
Dec 29 22:58:08 pve212 corosync[961]: [MAIN ] Completed service synchronization, ready to provide service.
Dec 29 23:01:19 pve212 corosync[961]: [TOTEM ] Token has not been received in 22 ms
Dec 29 23:04:37 pve212 corosync[961]: [TOTEM ] Token has not been received in 11 ms
Dec 29 23:10:02 pve212 corosync[961]: [TOTEM ] Token has not been received in 35 ms
Dec 29 23:10:02 pve212 corosync[961]: [TOTEM ] A processor failed, forming new configuration.
Dec 29 23:10:03 pve212 corosync[961]: [TOTEM ] A new membership (1.43a21) was formed. Members
Dec 29 23:10:03 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 23:10:03 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 23:10:03 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 23:10:03 pve212 corosync[961]: [QUORUM] Members[3]: 1 2 3
Dec 29 23:10:03 pve212 corosync[961]: [MAIN ] Completed service synchronization, ready to provide service.
Dec 29 23:17:01 pve212 CRON[57017]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Dec 29 23:20:19 pve212 corosync[961]: [TOTEM ] Token has not been received in 36 ms
Dec 29 23:20:19 pve212 corosync[961]: [TOTEM ] A processor failed, forming new configuration.
Dec 29 23:20:20 pve212 corosync[961]: [TOTEM ] A new membership (1.43a25) was formed. Members
Dec 29 23:20:20 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 23:20:20 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 23:20:20 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 23:20:20 pve212 corosync[961]: [QUORUM] Members[3]: 1 2 3
Dec 29 23:20:20 pve212 corosync[961]: [MAIN ] Completed service synchronization, ready to provide service.
Dec 29 23:27:11 pve212 smartd[661]: Device: /dev/sdc [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 66 to 67
Dec 29 23:27:11 pve212 smartd[661]: Device: /dev/sdc [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 34 to 33
Dec 29 23:29:08 pve212 corosync[961]: [TOTEM ] Token has not been received in 11 ms
Dec 29 23:29:09 pve212 corosync[961]: [TOTEM ] A new membership (1.43a29) was formed. Members
Dec 29 23:29:09 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 23:29:09 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 23:29:09 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 23:29:09 pve212 corosync[961]: [QUORUM] Members[3]: 1 2 3
Dec 29 23:29:09 pve212 corosync[961]: [MAIN ] Completed service synchronization, ready to provide service.
Dec 29 23:31:11 pve212 corosync[961]: [TOTEM ] Token has not been received in 6 ms
Dec 29 23:44:05 pve212 corosync[961]: [KNET ] link: host: 2 link: 0 is down
Dec 29 23:44:05 pve212 corosync[961]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Dec 29 23:44:05 pve212 corosync[961]: [KNET ] host: host: 2 has no active links
Dec 29 23:44:06 pve212 corosync[961]: [TOTEM ] Token has not been received in 35 ms
Dec 29 23:44:06 pve212 corosync[961]: [TOTEM ] A processor failed, forming new configuration.
Dec 29 23:44:06 pve212 corosync[961]: [KNET ] rx: host: 2 link: 0 is up
Dec 29 23:44:06 pve212 corosync[961]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Dec 29 23:44:06 pve212 corosync[961]: [TOTEM ] A new membership (1.43a2d) was formed. Members
Dec 29 23:44:06 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 23:44:06 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 23:44:06 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 23:44:06 pve212 corosync[961]: [QUORUM] Members[3]: 1 2 3
Dec 29 23:44:06 pve212 corosync[961]: [MAIN ] Completed service synchronization, ready to provide service.
Dec 29 23:45:23 pve212 corosync[961]: [TOTEM ] Token has not been received in 19 ms
Dec 29 23:45:24 pve212 corosync[961]: [TOTEM ] A processor failed, forming new configuration.
Dec 29 23:45:24 pve212 corosync[961]: [TOTEM ] A new membership (1.43a31) was formed. Members
Dec 29 23:45:24 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 23:45:24 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 23:45:24 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 23:45:24 pve212 corosync[961]: [QUORUM] Members[3]: 1 2 3
Dec 29 23:45:24 pve212 corosync[961]: [MAIN ] Completed service synchronization, ready to provide service.
Dec 29 23:51:35 pve212 corosync[961]: [KNET ] link: host: 2 link: 0 is down
Dec 29 23:51:35 pve212 corosync[961]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Dec 29 23:51:35 pve212 corosync[961]: [KNET ] host: host: 2 has no active links
Dec 29 23:51:36 pve212 corosync[961]: [TOTEM ] Token has not been received in 49 ms
Dec 29 23:51:37 pve212 corosync[961]: [KNET ] rx: host: 2 link: 0 is up
Dec 29 23:51:37 pve212 corosync[961]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Dec 29 23:51:37 pve212 corosync[961]: [TOTEM ] A new membership (1.43a35) was formed. Members
Dec 29 23:51:37 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 23:51:37 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 23:51:37 pve212 corosync[961]: [CPG ] downlist left_list: 0 received
Dec 29 23:51:37 pve212 corosync[961]: [QUORUM] Members[3]: 1 2 3
Dec 29 23:51:37 pve212 corosync[961]: [MAIN ] Completed service synchronization, ready to provide service.
Dec 29 23:52:33 pve212 corosync[961]: [KNET ] link: host: 2 link: 0 is down
Dec 29 23:52:33 pve212 corosync[961]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Dec 29 23:52:33 pve212 corosync[961]: [KNET ] host: host: 2 has no active links
Dec 29 23:52:35 pve212 corosync[961]: [TOTEM ] Token has not been received in 11 ms
Dec 29 23:52:35 pve212 corosync[961]: [KNET ] rx: host: 2 link: 0 is up
Dec 29 23:52:35 pve212 corosync[961]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Dec 29 23:57:12 pve212 smartd[661]: Device: /dev/sdc [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 67 to 66
Dec 29 23:57:12 pve212 smartd[661]: Device: /dev/sdc [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 33 to 34
Dec 30 00:00:00 pve212 systemd[1]: Starting Rotate log files...
Dec 30 00:00:00 pve212 systemd[1]: Starting Daily man-db regeneration...
Dec 30 00:00:00 pve212 pvefw-logger[651]: received terminate request (signal)
Dec 30 00:00:00 pve212 pvefw-logger[651]: stopping pvefw logger
Dec 30 00:00:00 pve212 systemd[1]: Stopping Proxmox VE firewall logger...
Dec 30 00:00:00 pve212 systemd[1]: pvefw-logger.service: Succeeded.
Dec 30 00:00:00 pve212 systemd[1]: Stopped Proxmox VE firewall logger.
Dec 30 00:00:00 pve212 systemd[1]: Starting Proxmox VE firewall logger...
Dec 30 00:00:00 pve212 systemd[1]: man-db.service: Succeeded.
Dec 30 00:00:00 pve212 systemd[1]: Started Daily man-db regeneration.
Dec 30 00:00:00 pve212 pvefw-logger[61435]: starting pvefw logger
Dec 30 00:00:00 pve212 systemd[1]: Started Proxmox VE firewall logger.
Dec 30 00:00:01 pve212 CRON[61445]: (root) CMD (vzdump 120 300 --node pve211 --storage local211 --mailnotification always --quiet 1 --compress zstd --mode snapshot)
Dec 30 00:00:01 pve212 CRON[61443]: (root) CMD (vzdump 140 --compress zstd --quiet 1 --node pve210 --storage local211 --mailnotification always --mode snapshot)
Dec 30 00:00:01 pve212 CRON[61444]: (root) CMD (vzdump 400 140 --mode snapshot --mailnotification always --node pve212 --storage local212 --compress zstd --quiet 1)
