wipefs error probing initialization failed

Wipefs error probing initialization failed serial numbers#

Now they don't bomb when zapping, but they remain in that state and can't be added. S1 /dev/sda ssd Samsung_SSD_860_EVO_1TB_S32T 1000G Insufficient space (<10 extents) on vgs, LVM detected, locked Right after rebooting (after setting unmanaged=true), the disks show up thus: The fact that the devices are locked led me to believe that somehow the management was competing with itself trying to bring them up as OSDs. usr/bin/podman: stderr -> failed to wipefs device, will try again to workaround probable race condition usr/bin/podman: stderr stderr: wipefs: error: /dev/sdb: probing initialization failed: Device or resource busy usr/bin/podman: stderr -> Zapping: /dev/sdb This spits out some output, and this seems to indicate that the devices are actually, well, locked: Running ceph orch device zap tries to do lvm zap -destroy /dev/sdb, which in turn calls wipefs. ceph orch apply osd -all-available-devices -unmanaged=true (and then rebooting everything again) Even rebooting all of the nodes at the same time. I used sgdisk -Z, and ceph orch device zap s1 /dev/sdb -force.

Wipefs error probing initialization failed serial numbers#

(names and serial numbers shortened for clarity and privacy) S3 /dev/sdb ssd Samsung_SSD_860_EVO_1TB_S33M 1000G locked S3 /dev/sda ssd Samsung_SSD_860_EVO_1TB_S36V 1000G locked S2 /dev/sdb ssd Samsung_SSD_860_EVO_1TB_S35A 1000G locked S2 /dev/sda ssd Samsung_SSD_860_EVO_1TB_S35E 1000G locked S1 /dev/sdb ssd Samsung_SSD_860_EVO_1TB_S37M 1000G locked S1 /dev/sda ssd Samsung_SSD_860_EVO_1TB_S32T 1000G locked HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS This is how they appear to the first node: I took a few SSDs off other clusters, wiped them and tried to add them. The cluster is up and mostly healthy but warning that it doesn't have any OSDs. Ceph is currently at 16.2.7+ds-4+b2, which is the same version string as for cephadm. I already shutdown all services except systemd, ssh, polkit, rsyslog and mdadm monitor.I have installed a cluster with 3x Dell r730 servers using Debian Bookworm's cephadm. Currently I do not have physical access to the machine.

I can't restart the server, because the system isn't bootable now. I'm not using LVM, no userspace is installed: ~ # vgsĬommand 'vgs' not found, but can be installed with: Jbd2/md2- 599 root txt unknown ~ # fuser -v ~ # fuser -v /dev Md2_raid1 262 root txt unknown /proc/262/exe Mdadm: set device faulty failed for /dev/nvme0n1p2: Device or resource busyįuser or lsof does now shoe any process except kernel threads: ~ # lsof | grep md2 Mdadm: hot removed /dev/nvme1n1p2 from ~ # mdadm /dev/md2 -fail /dev/nvme0n1p2 -remove /dev/nvme0n1p2 Mdadm: set /dev/nvme1n1p2 faulty in /dev/md2 Mdadm -add /dev/md2 ~ # mdadm /dev/md2 -fail /dev/nvme1n1p2 -remove /dev/nvme1n1p2

Mdadm: set device faulty failed for /dev/nvme1n1p2: Device or resource busy Mdadm: hot removed /dev/nvme0n1p2 from ~ # mdadm /dev/md2 -fail /dev/nvme1n1p2 -remove /dev/nvme1n1p2 Mdadm: set /dev/nvme0n1p2 faulty in /dev/md2 Mdadm: Cannot get exclusive access to /dev/md2:Perhaps a running process, mounted filesystem or active volume ~ # mdadm /dev/md2 -fail /dev/nvme0n1p2 -remove /dev/nvme0n1p2 I also can't stop the mdam array to set the while raid array to fail: ~ # mdadm -stop /dev/md2 -force dev/md2 is apparently in use by the system will not make a filesystem here! Last mounted on /boot on Fri Sep 18 21:33:34 2020 dev/md2 contains a ext4 file system labelled 'boot' Wipefs: error: /dev/md2: probing initialization failed: Device or resource ~ # mkfs.ext4 -L boot /dev/md2 I'm unable to reformat or destroy the partition: ~ # wipefs /dev/md2 Touch: cannot touch '/boot/a': Structure needs cleaningĭmesg shows me: EXT4-fs error (device md2): ext4_find_dest_de:1932: inode #2: block 518: comm touch: bad entry in directory: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0, size=1024Įven after successfully umount the device, I can't fsck ~ # umount ~ # e2fsck /dev/md2Į2fsck: Cannot continue, ~ # fsck.ext4 /dev/md2 Command-line wipefs -a '/dev/sdb' exited with non-zero exit status 1: wipefs: error: /dev/sdb: probing initialization failed: Device or resource busy (udisks-error-quary,0) Nothing I know of us using it and I am doing this from a Live CD boot. While is normally no problem due backups, I can't write to the partition anymore.