| ikonia | is the snap ubuntu core-18 actually providing part of the operating / minimal file system, I can't work out the overlap between what ubuntu core is providing against what the debs are providing against the basic ability to boot the machine | 08:23 |
|---|---|---|
| ikonia | on the 'broken' pi vmstat shows large wait 'wa' on io yet I can't see any actual file system corruption just really slow response | 08:25 |
| rbasak | Sounds like you're not using Snappy Core, but just the regular install? In that case, if you're not using specific snaps, then you don't need snapd or core18. | 10:27 |
| rbasak | snapd might be updating the core18 image on a regular basis which will cost writes. | 10:27 |
| rbasak | OTOH, there's a whole other thing, where everything is a snap, there are no debs, and the system is read-only. That'll help with reducing writes, but I don't think you're operating that way? | 10:28 |
| ikonia | it's not the snap IOT release, standard ubuntu pi build from the rebuilt image | 10:55 |
| ikonia | I've managed to be %99 sure that this SD card is failing/failed - this is the 4th one I think, so clearly something is eating these SD cards up | 10:56 |
| ikonia | I think based on the requirements of consul, I'll look into netbooting the pi and using ram disks as a very small install with consul and puppet installed and held in ram and logging central than locally will probably be a far better solution than to keep eating these SD cards up | 10:57 |
| waveform | ikonia, do all the SD cards come from the same buy? (i.e. a bulk purchase -- just wondering if batch-failure is a possible cause here as I've encountered this once with a batch of kingston cards) | 10:57 |
| ikonia | waveform: nah, spread out purcahses, but all sandisk (official shop not dodgy copy) | 10:57 |
| ikonia | eg: the one that's just failed was purcahsed a few weeks ago to repalce one that had already failed so ~4 months apart | 10:58 |
| waveform | ikonia, hmm -- it does sound odd. Out of curiosity what sort of pi is this? | 11:02 |
| ikonia | waveform: pi 4 | 12:32 |
| ikonia | dmesg now showing signs that the card is failing | 12:33 |
| ikonia | [60596.408352] blk_update_request: I/O error, dev mmcblk0, sector 2697408 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0 | 12:33 |
| ikonia | that wasn't there yesterday | 12:33 |
| ikonia | I a curious to if any of you arm enthusiats are net booting the pi's | 12:34 |
| ikonia | either with the pi image or core | 12:34 |
| ikonia | it feels like that will solve a lot of the problems for the use case I've got | 12:34 |
| waveform | ikonia, okay -- in that case netbooting may well be a decent option (the gigabit eth makes a major difference to networking performance compared to prior models) | 12:34 |
| ikonia | waveform: exactly, from what I've read so far, local SD card to boot the firmware and that's it the rest is network | 12:35 |
| ikonia | plus if most of the data is transiet, then commit to disk over network will be slim | 12:35 |
| waveform | ikonia, you can netboot it purely without an SD card at all if you want | 12:35 |
| ikonia | waveform: no way ! I'd not read that yet | 12:35 |
| waveform | ikonia, if you're using groovy (20.10) onwards, you can netboot it by following the usual raspbian instructions (since we use the "typical" raspbian boot sequence on those images). On focal (20.04) you can modify the image to use the "typical" boot sequence (permitting sd-card-less netbooting) with a bit of surgery -- I'll dig out my instructions | 12:36 |
| waveform | (which are actually for USB booting, but it's all the same since it basically involves excising u-boot from the sequence) | 12:36 |
| waveform | here we go: https://waldorf.waveform.org.uk/2021/you-boot-no-u-boot-first.html | 12:36 |
| ikonia | let me have a read, I'm on 20.04 as I value the stabiliy of the tools around it | 12:36 |
| waveform | fair enough | 12:37 |
| ikonia | that would be amazing to pure netboot it, I'd not considered that's an option | 12:37 |
| waveform | yup, once you modify the focal image to remove u-boot from the sequence, just follow the usual raspbian netboot instructions (just a mo, I'll dig those out) | 12:37 |
| waveform | https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/net_tutorial.md | 12:37 |
| ikonia | this has certainly excited me | 12:37 |
| ikonia | have you done this / are you doing it ? | 12:37 |
| ikonia | (just reached for my test pi to get moving) | 12:38 |
| waveform | yup, I've netbooted a pi400 into the groovy desktop -- it doesn't perform as well as booting a USB SSD but that's no surprise :) | 12:38 |
| ikonia | I guess if I thin the image down as much as possible there is a possible the whole OS could be held in uncompressed ram | 12:39 |
| waveform | and I've netbooted a pi3 into the groovy server as well (just to test it worked). I can't say I've tested it since but that's because the boot sequence hasn't changed, and I mostly stick to booting from SD card (for image testing and a few low-load pis here), or USB (for my development pis) | 12:39 |
| ikonia | the consul service I use them for is very slim | 12:39 |
| ikonia | exciting stuff ! | 12:40 |
| waveform | on the larger pi4s there's likely enough RAM for that (especially the 8GB model) -- can't say I've tried it, but it's probably possible with some effort | 12:40 |
| ikonia | I've got a few 4gb test boxes here, if it works in principal I can move it to the 8gb models | 12:41 |
| ikonia | quite excited by this as a concept | 12:41 |
| ikonia | if it boots a usb with an ssd, or even spinning disk, that may even solve the problem of the SD card's failing | 12:43 |
| waveform | certainly worth considering -- booting of a USB SSD provides the best performance but be warned there is an issue with *some* USB SSDs seemingly resetting at some points. We haven't gotten to bottom of this yet and some people never encounter it while others see it frequently (I've seen it about once a day on one of my pi4s) | 12:44 |
| waveform | https://bugs.launchpad.net/ubuntu/+source/linux-raspi/+bug/1930629 for reference | 12:44 |
| waveform | and the upstream bug (with looooong commentary) is https://github.com/raspberrypi/linux/issues/3981 | 12:45 |
| waveform | anyway, certainly worth a try but don't rely on it for production until you've run it continuously for say a week without observing any USB port issues | 12:45 |
| ogra | ikonia, FYI (since you asked above), Ubuntu Core is completely built ut of snaps (kernel, bootloader, rootfs and all apps are snaps), 99% of the filesystem are readonly, writes to SD are massively reduced ... | 12:47 |
| ikonia | ogra: where does that fit into the ubuntu raspberry pi image | 12:49 |
| waveform | ikonia, ubuntu core and ubuntu server (and ubuntu desktop) for pi are all separate images; core is the "snap only, no debs" image | 12:50 |
| ikonia | yeah, the pi image seems to be made up of both | 12:50 |
| ogra | ikonia, it is a separate image .... not the same as ubuntu-server or an ubuntu-base tarball | 12:50 |
| waveform | the server image includes both debs and snaps | 12:50 |
| ikonia | hence why I'm not %100 sure | 12:50 |
| ogra | right, snapd is in all ubuntu default installs nowadays | 12:51 |
| ogra | but that only means you can install snap packages in all of them | 12:51 |
| ikonia | the pi image has snapd with core-18 and lxd already installed | 12:51 |
| ogra | in Ubuntu Core you can not install debs ... in -server and -desktop you can use both | 12:51 |
| ogra | yes, lxd is also in all default installs AFAIK ... and only available as a snap | 12:52 |
| waveform | the core image is also *much* smaller than server (which is a bonus for various use-cases), but I would caution that it still relies upon u-boot and doesn't currently support netboot (it might be able to with some work, but last time I tried, there was some major blocker to getting it working on the pi4s -- I forget exactly what) | 12:52 |
| ikonia | I wasn't %100 if the core snap was providing any of the root file system in the booted system | 12:53 |
| ogra | it provides the rootfs of snaps at runtime .... | 12:53 |
| ogra | snaps are running on top of their respective "base" snap (core, core18, cre20) | 12:53 |
| ikonia | ahhhhhhhhh | 12:53 |
| ikonia | ok | 12:53 |
| ikonia | yeah, like a container | 12:53 |
| ikonia | sorry I was being dumb | 12:53 |
| ogra | similar but diferent, yeah 🙂 | 12:54 |
| ikonia | I get it - I thought the snap may have been providing part of the running system file system | 12:54 |
| waveform | snapd can be removed from the server image assuming you're not relying on any snaps (but bear in mind that some default things like lxd are only shipped as snaps in recent releases) | 12:55 |
| ogra | nope, and you can just apt orged snapd if you do not want it | 12:55 |
| ogra | *apt purge snapd | 12:55 |
| ikonia | makez sense | 12:55 |
| * ogra glares at his fingers | 12:55 | |
| ikonia | this could open up a real potential on the pii's for me | 12:55 |
| ikonia | assuming the netboot performs ok | 12:55 |
| waveform | on the pi4s it's "okay". The gigabit eth on there makes a big difference but the real killer is the latency. If you're doing something involving IO across lots and lots of files (staring at apt/dpkg...) it gets slow even compared to an SD card because suddenly there's a ton of network roundtrips in your IO | 12:58 |
| ikonia | I think that will be where the balance is in the testing, | 12:59 |
| waveform | on the other hand, if your IO involves big reads/writes across a few files it's massively faster than the SD card -- so yes, it'll heavily depend on your particular workload | 12:59 |
| ikonia | I'm curious to have something like a consul node will actually working / perform against the need to keep the disk in sync over the network | 12:59 |
| ikonia | on paper my workload will fly | 13:00 |
| ikonia | I'm keen to see the truth of that | 13:00 |
| ogra | well, if you want it performant and have an 8GB pi, i'd remotely mount a squashfs root and use overlayfs with the writable part in RAM ... | 13:04 |
| ikonia | as almost all the data is transient, I don't think I even need that | 13:04 |
| ikonia | but I need to build my image first, see how small it really is | 13:04 |
| ikonia | interesting so NFS is the only real way to offer out the root file system when network booting | 14:41 |
| ikonia | that's a little unexpected | 14:42 |
| waveform | ikonia, currently, yes -- I'm aware of some people working on openiscsi alternatives, but nothing that definitely works in that area yet | 14:58 |
| ogra | nbd booting should work too as long as you have the correct set of modules in the initrd | 15:08 |
| ikonia | ooooh openscsi is exciting for a pi | 15:32 |
| ikonia | never done nbd | 15:34 |
| ikonia | all a bit of exciting learning, I'm not keen on the NFS approach just as NFS is old and can be troublesome | 15:36 |
| ikonia | clearning up the base install is a challenge in itself, 500+ packages is a big base to start with | 15:39 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!