[00:55] slangasek, if you're sitting there twiddling your thumbs, and you have any ideas on https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1611074 i'd like to hear them. [00:55] Launchpad bug 1611074 in cloud-init "Reformatting of ephemeral drive fails on resize of Azure VM" [High,Fix committed] [00:55] hard to type and twiddle at the same time [00:56] i ususally peck with my nose. [00:56] it wastes time that way too [00:57] slangasek, one thing that seems wrong... but i'm not sure its entirely related. [00:57] http://paste.ubuntu.com/23488127/ [00:57] when i run that script well after boot, sometimes i hit line 29.. that is to say systemd magically mounts the partition for me. [00:59] smoser: but in this context you're repartitioning it from a booted state? in which case the x-systemd.requires won't protect you [00:59] correct. [00:59] but i'd still find it odd and wrong to umount anad then have something magically mount for me. [01:01] and i'im not convinced that its not related. [01:01] most certainly, systaemd generator writes those files based on the stale fstab [01:01] it doesn't seem incorrect for systemd to automount for you a device that it discovers via udev, post-boot, whose deps are satisfied [01:01] it seems to me that that should be a 'once' thing. [01:02] no other mount service continually mounts automatically for you. [01:02] unless there is .automount unit; or udev got retriggered for the device; "stop" the .mount unit manually? [01:02] especially wrong when it could be mounting garbage of a filesystem that just happend to be at the place where you put the new partition. [01:02] xnox: in this context, udev has gotten retriggered for the device yes [01:03] check the generator and the .automount / .mount units in /run/systemd ?! [01:03] smoser: so don't label your garbage with the exact same name that you were using to mount it by in /etc/fstab? [01:03] because this is the exact same code used to automount a removable device when you plug it in [01:04] which you might remove, and want to mount again [01:04] use $ wipefs -a -> to wipe garbage off the partition? [01:04] slangasek, you've not gotten a chance at that point to write a filesystem. [01:04] and xnox your case doesn't even help... its really busted. [01:04] yes, you can be careful and get aroudn this [01:04] but it is nonsense. [01:04] ie, if you want to repartition a disk, you have to: [01:04] smoser: yes, because your fstab refers to the partition in a way that triggers the mount before you've created a filesystem [01:04] a.) unmount everythin [01:05] b.) seek to the new partition table offsets and wipe any filesystem stuff that might possibly be there. [01:05] so if you're expecting to be able to do this online, post-boot... don't refer to the filesystem by a device name that shows up before the filesystem is created? [01:05] c.) partition it [01:05] d.) mkfs the partition [01:06] you have to do 'b' because where you put the new partition might have had a valid filesystem signature. even from an attack of some sort. [01:07] wipefs does support offset argument -o [01:07] interestingly, udev in all its auto-glory watches rw opens on /dev/devices [01:07] smoser, i had hallarious bug reports in d-i w.r.t. serial re-installers in virtual machines, that use host LVM volumes as backing device stores. [01:07] and rescans partitiontables for you automatically when wipefs closes [01:07] so... lots of fun. [01:08] right. this magic is quite nonhelpful when you dont want it. [01:08] smoser, because create a fresh volume, may just align with old volume, and give you perfectly valid filesystem signatures, to bust the installations in funny ways. [01:08] thats what im' saying [01:08] *and* that data could actually be an attack [01:08] if you don't want it, that's called 'noauto' [01:09] in your /etc/fstab entry, yes. [01:09] which was stale [01:09] and is still magically present when you remove it [01:09] how is it stale [01:09] because /etc/fstab isn't the thing that is doing these mounts [01:09] ? [01:09] its stale because its present from before the stop [01:09] systemd-fstab-generator [01:09] and then the new disk attached. [01:09] it's stale /with respect to what/? [01:10] your example script doesn't show any editing of /etc/fstab [01:10] so unmount; remove ftab; systemctl daemon-reload (which should retrigger generators, and remove stale .mount generated units); do things; write new ftab; mount things ? [01:10] sure. but that doesnt matter, slangasek. [01:10] it shows systemd following the instructions it's been given, faster than you're expecting [01:10] i can do it, it doesnt stop the .mount service [01:10] no one expects to edit /etc/fstab and stop a service to partition a disk. [01:11] (i have specifically taken the line out of /etc/fstab) [01:11] so yes, best case what you're suggesting is that i have to [01:11] a.) edit /etc/fstab [01:11] b.) systemctl daemon-reload [01:11] c.) partition , mkfs [01:11] d.) edit /etc/fstab [01:11] e.) mount [01:12] and 'b'probably doesnt work during boot [01:12] i dont know that, but i suspect as many systemd things do not [01:13] isn't the *actual* problem you're trying to solve that systemd should not automount the disk at boot time, before cloud-init has run? [01:15] if so, let's please solve that problem, and not have a proxy battle about how automounting is designed in systemd [01:15] if not, I don't understand the scope of the bug so please explain it to me [01:15] yes thats the problem. but i think it is related to the magic that i'm seeing. [01:15] its possible its not. [01:16] but essentially, cloud-init is doing the same thing as the script during boot, and then failing to mkfs because something has mounted the device. [01:16] i' not certain its related to this, but it seems similar. [01:17] /rules.d/80-iio-sensor-proxy.rules [01:17] haaate [01:18] smoser: in the case where it has failed at boot, what are the contents of /run/systemd/generator/mnt.mount ? [01:19] i'll recreate. it'll take me a bit. [01:19] anything else you might want from before the failed boot ? [01:21] smoser: I don't know well enough what all I would want; I would really want to debug it interactively [01:24] slangasek, i can let you in before and after. [01:25] slangasek, my attempts to simplify fail [01:25] ie, i've booted system... then just partitioned the existing disk and put ntfs on partition 1 [01:25] and then rebooted [01:25] which is much the same as stop, new ephemeral diska ttached, start [01:26] but i believe the start after resize has slower disks at first [01:26] and so we hit it there but not on the "warm" reboot disks [01:26] at least i've not seen it my way [01:38] slangasek, if you want ssh smoser@40.122.215.16 [01:38] that is prior to resize [01:38] smoser: and without an up-to-date /etc/fstab? [01:38] /dev/disk/cloud/azure_resource-part1 /mnt auto defaults,nofail,comment=cloudconfig 0 2 [01:38] no modifications to that. [01:38] yeah [01:39] i saved a bunch of stuff in first-boot [01:39] so you'll be manually changing that before resizing? [01:39] we can reboot ( which will do that) [01:39] or we can manually do it [01:39] i'm apt to reboot, and collect those logs and state and such [01:39] and then resizse [01:40] so far, launch, add proposed && install cloud-init , collect some info [01:41] smoser: if you rely on fstab being updated via reboot, then indeed you could be fighting with the systemd fstab generator [01:41] oh [01:41] you mean reboot once, to regenerate fstab; then reboot again for the resize? [01:41] right. [01:42] yeah, +1 [01:43] ok. rebooted, ran 'save-old-data second-boot' [01:43] and [01:43] /dev/disk/cloud/azure_resource-part1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2 [01:44] and now api resize [01:46] smoser: right - fstab looks good, /run/systemd/generator/mnt.mount still wrong but that's as expected [01:46] wrong ? [01:47] smoser: /run/systemd/generator/mnt.mount was generated by systemd-fstab-generator before cloud-init updated /etc/fstab, therefore it has the wrong dependencies [01:47] right. [01:51] slangasek, ok. its back up, and it reproduced [01:51] and, slangasek, thank you for your help. [01:52] smoser: ok. 'systemctl status mnt.mount' told me 'Warning: mnt.mount changed on disk. Run 'systemctl daemon-reload' to reload unit' [01:53] smoser: but 'systemctl show mnt.mount' does show the correct dependencies for the running unit [01:54] right. [01:55] smoser: 'systemctl list-dependencies' makes me somewhat suspicious, mnt.mount isn't shown as depending on cloud-init.service [01:55] and nothing actually changed in the /etc/fstab, its just that systemd recognized probably a timestamp cahnge in /etc/fstab and says its out of date. [01:55] slangasek, it does show it [01:56] at least [01:56] systemctl list-dependencies mnt.mount [01:56] does [01:56] yes [01:58] thats interesting [01:58] smoser: and cloud-init.service is where the resize happens. And cloud-init.service is Type=oneshot. [01:58] right. [01:58] so After=cloud-init.service is basically "start this as soon as you've exec()ed the other one" [01:58] systemd.service(5), Type= [01:59] e [01:59] er [01:59] no, sorry [01:59] that's completely wrong, I should read more carefully teh description of oneshot ;) [02:00] hmm. [02:00] RemainAfterExit is odd though [02:00] wrt oneshot and requires [02:01] "RemainAfterExit= is particularly useful for this type of service" [02:01] s/useful/confusing/ [02:01] what would that even mean.. if the thing remains, you could essentially never be After [02:01] smoser: it means that it's listed as 'active', but no changes to the behavior of After [02:02] that makes more sense. [02:02] smoser: '/usr/bin/cloud-init init' wouldn't background for any reason, would it? [02:02] well... it could start some processes [02:02] that backgrounded [02:02] but it wont. [02:03] so the top-level cloud-init process is still running while the partitioning/mkfs is happening [02:03] yeah [02:03] but it might start a daemon of some sort [02:05] smoser: why does 'systemctl status cloud-init' show the job's last status change at 01:49:06, but the last journal output entries at 01:49:07? [02:05] try monotnoic [02:05] the clock bounces all over the place on those system.s [02:05] heh [02:06] but i dont know if you can get monotonic on status [02:06] can you ? [02:06] doesn't appear so [02:06] but if I compare stamps from mnt.mount and cloud-init.service, mnt.mount is definitely after [02:07] smoser: and the timestamp shown in /var/log/cloud-init.log for mkfs.ext4 is 2016-11-17 01:49:09, which is definitely later than what was reported in the journal for either job... so seems less likely to be due to a clock bounce [02:08] journalctl -o short-monotonic --no-pager --full [02:08] and you can see [02:08] [ 256.905798] reprovm systemd[1]: Time has been changed [02:09] so one really wierd thing that happens on azure... [02:09] we bounce eth0 (ifdown ; ifup) [02:09] because thats how you publish your hostname (if the user modified it) to the platform [02:10] i dont think its related... but it means dhcp hooks and possibly time jumps and lots of general wierdness. [02:10] so i'm just mentioning [02:10] smoser: well, hard to be sure without monotonic log output for cloud-init.log itself then [02:11] well, you can get it.. [02:11] from journalctl [02:11] as cloud-init writes to journal its stdout [02:11] * slangasek looks [02:11] so the time obviously differs by when it got seen versus when it got written [02:11] but, kind of no way to avoid that [02:12] smoser: well, journalctl -u cloud-init.service is only showing me a handful of lines, not all the output in /var/log/cloud-init.log [02:12] hm.. [02:13] just journalctl -o short-monotnoic [02:13] i see [02:13] [ 23.003468] reprovm systemd[1]: Mounted /mnt. [02:13] smoser: and the output shown in journalctl -u cloud-init.service also does not appear in /var/log/cloud-init.log [02:14] right. [02:14] because it is set to write debug to log [02:14] but less verbose to stdout [02:14] we can cahnge that and try again [02:14] smoser: actually, this log did finally show me what I needed [02:14] with -o short-monotonic [02:14] [ 22.998682] reprovm systemd[1]: Started Initial cloud-init job (metadata service crawler). [02:15] [ 24.812954] reprovm cloud-init[1077]: Failed to exec of '['/sbin/mkfs.ext4', '/dev/sdb1']': [02:15] the job was marked 'started' 2 seconds before the mkfs was run [02:15] so cloud-init does somehow seem to be backgrounding this [02:16] rather, this doesn't look at all like it's running mkfs from the cloud-init.service [02:16] but from cloud-config.service [02:16] so... wrong dep? [02:18] well, cloud-init.service is supposed to be running them. [02:18] is there a way to tell journalctl not to rotate [02:19] whenver i give it fire hose, i find later that it can't show me stuff saying its rotated [02:20] you're right, it certainly does seem to be running fromcloud-config, but it should not be. [02:20] it's configurable, i don't recall how. you can also touch the file to get an on-disk journal, which we don't do by default, then you don't have to rotate to avoid running out of memory [02:21] ok [02:21] oh. [02:21] but I'm guessing you can take it from there to figure out why it's being run from the wrong part of cloud-init :) [02:21] yeah. thank you. [06:42] good morning [07:26] Good morning [07:36] pitti: h [07:36] i [07:36] pitti: FYI on the gnutls thing I mentioned hitting libvirt yesterday [07:36] pitti: it seems we need to revert that change in gnutls instead of adapting libvirt [07:37] pitti: I added a gnutls task to the bug I was working on and currently prepping an upload for a ppa to try it out [07:37] pitti: that was bug 1641615 [07:37] bug 1641615 in libvirt (Ubuntu) "FTBFS of libvirt 2.1 in zesty" [High,Confirmed] https://launchpad.net/bugs/1641615 [07:52] cpaelzer: that's only gnutls30, right? I thought 28 was still the default (not sure if there's a transition going on) [07:54] pitti: I must admit the gnutls package versioning is hard at first so IÄm not sure I get it completely yet [07:54] pitti: it is libgnutls30 with headers in libgnutls28-dev out of source gnutls28 [07:54] pitti: did that make sens to you? [07:54] not off-hand -- I haven't looked into this at all [07:55] pitti: the changelog says you did the merge which is why I was highlighting you :-) [07:56] oh right, this isn't a transition; it's just weird naming [08:16] anybody that may help me with my ubumirror problem? hard to offer a mirror server to the community if the actual mirror scripts don't work as expected :D [08:19] hi tuxinator I've seen your request like three times but have no experience around that [08:19] tuxinator: but I really appreciate the support willing to provide a mirror - maybe the right people are not here on the itme you are asking ... ? [08:20] tuxinator: have you tried asking at https://lists.ubuntu.com/mailman/listinfo/ubuntu-mirrors ? [08:21] tuxinator: there also seems to be a #ubuntu-mirrors IRC channel on freenode - thou I'm not sure on that one [08:21] surely there the request won't be drowned in so many other messages [08:24] tuxinator: there also is mirrors@ubuntu.com - not sure which mail is for the offical mirrors and which for mirror clones of any sort [09:51] pitti: has systemd unit masking (i.e. linking the unit to /dev/null) a recent introduction in systemd ? [09:53] caribou: no, that has existed basically forever [09:54] (for sure "forever" in ubuntu) [09:54] pitti: ok thanks just wanted to be sure [10:13] TheMuso: I don't understand why you dropped pulseaudio-equalizer. Yes, its deps are in universe, but it's perfectly fine for a source in main to produce a binary in universe. [10:14] TheMuso: Seems like an entirely pointless delta from Debian. [11:42] LocutusOfBorg: please file a bug against the git-build-recipe project about that [11:42] tuxinator: I asked you a question to try to narrow down your problem but you didn't reply [11:44] cjwatson, probably I was disconnected [11:44] will look to logs [11:44] pitti: on bug 1641615 I created a gnutls fix - since you made the merge could you take a look at reviewing and sponsoring that? [11:44] bug 1641615 in libvirt (Ubuntu) "gnutls api breakage in 3.5.6" [High,Confirmed] https://launchpad.net/bugs/1641615 [11:45] oh was directed to tuxinator the second sentence [11:46] cpaelzer: ack, queueing === _salem is now known as salem_ [11:50] pitti: thanks and if possible keep me up to date so I can follow with the matching libvirt un-fix which is needed now (or push that along if you e.g. push that via bileto - its debdiff is on the bug as well) [11:58] cpaelzer: thanks [11:58] cpaelzer: actually there is no hurry :D === hikiko is now known as hikiko|ln === JanC_ is now known as JanC === hikiko|ln is now known as hikiko [13:50] pitti: hi! are we going to get postgresql-9.6 in zesty, or should I revert the pgmemcache package back to supporting 9.5? [14:01] pitti: I'll just revert it for now so I can get my other stuff migrated [14:07] mdeslaur: I haven't thought much about this; TBH, I think we should avoid major version bumps in the non-LTS releases, as that simplifies maintenance [14:07] and noone in their right mind would use a 9-month release for a server/production setup anyway [14:07] mdeslaur: so I was going to leave it at 9.5 until people complain loudly :) [14:07] pitti: sounds good, thanks! [14:22] if I wanted to fix a bug in an ubuntu package file (the .dsc file) where would I go to submit that change? [14:22] cpaelzer: this is all backported from trunk, right? [14:23] pitti: yes [14:23] pitti: the git ids are from gnutls [14:23] cpaelzer: note, as these symbols are not really from upstream 3.5.6, .symbols must be stricter [14:23] + gnutls_ocsp_resp_get_responder2@GNUTLS_3_4 3.5.6-4ubuntu2~ [14:23] cpaelzer: ^ like this [14:23] pitti: true, but there is no 3.5.7 nor a 3.5 release [14:23] cpaelzer: (fixing here, just OOI) [14:23] s/OOI/FYI/ [14:24] TLAs are hard! [14:25] cpaelzer: ack, thanks; uploaded [14:25] cpaelzer: either reupload libvirt after it built/published everywhere, or bump the build dep to that version === cpaelzer_ is now known as cpaelzer [14:38] rbasak: If I want to request a package be added to a packageset, I should send an email to devel-permissions@lists.u.c and then add it to the agenda for an upcoming DMB meeting, right? [14:40] Odd_Bloke: correct. It shouldn't be a requirement to add it to the agenda, but no harm if you do - if anything it gives us a place to assign it to someone to deal with if it doesn't get looked at before a meeting. === foxtrot is now known as tex === tex is now known as texas [15:28] anyone know a trick to get a foreign build-dep to work within a cross-compile chroot? (eg, apt-get build-dep :arm64) [15:29] it always seems to get tangled up trying to install and run foreign packages (eg, python) === texas is now known as foxtrot === JanC is now known as Guest90855 === JanC_ is now known as JanC === alan_g is now known as alan_g|tst === alan_g|tst is now known as alan_g [16:54] doko: i think i figured out the freeradius issue; will send to debian and z-p [18:05] so freeradius has a /var/log/freeradius/wtmp and /var/log/freeradius/utmp file. One of it's autopkgtests just runs `radlast` which looks at that wtmp file. However, by default, that file doesn't see to exist. Normally, what would be responsible for creating that file? A debian postinst step? The freeradius daemon itself? [18:06] *see -> seem [18:46] ah i think i see what is supposed to happen -- the 'unix' driver creates the wtmp file on the first accounting with freeradius [18:46] not sure how that worked in 16.04 w/ the same tests, will go investigate [18:56] ah ha, in 16.04, postinst created the files [19:11] if I wanted to fix a bug in an ubuntu package file (the .dsc file) where would I go to submit that change? [19:15] bswartz: which package? [19:15] bswartz: it sort of depends, but the normal way for a user without upload rights to submit, is to open a bug, and put a debdiff in the bug [19:16] nacc: xfce4-clipman-plugin [19:17] nacc: I did open a bug but I'm not 100% it was in the right place [19:17] bswartz: bug #? [19:17] https://bugs.launchpad.net/ubuntu/+source/xfce4-clipman-plugin/+bug/1641802 [19:17] Launchpad bug 1641802 in xfce4-clipman-plugin (Ubuntu) "xfce4-clipman-plugin should depend on libqrencode-dev" [Undecided,New] [19:26] bswartz: have you worked on debian or ubuntu packages before? (just trying to konw where to start to help you out) [19:27] nacc: I've hacked on all kinds of things -- modifying .dsc files seems to be particularly easy [19:27] I'm not sure what the offcial workflow is for testing a change though [19:27] bswartz: right, you generally wouldn't chagne the .dsc file directly [19:27] In my case I simply couldn't stand my eye's bleeding at how bad the xfce4-clipman plugin in Ubuntu is compared to Fedora, so I decided to fix it for myself [19:28] bswartz: so here's my suggestion just to get you going: [19:28] in an appropriate directory: pull-lp-soure -cd xfce4-clipman-plugin [19:28] bah [19:28] -d (not -cd) [19:28] and pull-lp-source [19:29] and actually, in your case, just pull-lp-source (no -d flag either) [19:29] so let me start over! :) [19:29] pull-lp-source xfce4-clipman-plugin [19:29] cd xfce4-clipman-plugin- [19:29] make some changes [19:29] dch -i [19:29] describe your changes and put the right version in [19:30] (as well as targeting the correct release) [19:30] dpkg-buildpackage -S -nc -d -uc -us [19:30] cd .. [19:31] debdiff xfce4-clipman-plugin_.dsc xfce4-clipman-plugin_.dsc | less [19:31] (review the diff) [19:31] if it looks good, you can do: [19:31] pull-lp-source: Warning: Distribution data outdated. Please check for an update for distro-info-data. See /usr/share/doc/distro-info-data/README.Debian for details. Or specify a distribution. [19:31] I'm on xenial [19:32] debdiff xfce4-clipman-plugin_.dsc xfce4-clipman-plugin_.dsc | lp-attach [19:32] bswartz: hrm, are you up-to-date? [19:32] I thought I was [19:32] bswartz: i thought they had rolled out hte zesty data to the older distributions [19:35] turns out some things did need updating, I'm dist-upgrading now [19:35] bswartz: ok, that will probably just make xenial aware that zesty exists (so it can recognize the name, etc) [19:36] doko: jgrimm: freeradius autopkgtests pass now, uploaded and sent the fix to debian [19:40] thanks nacc [19:41] nacc: now I am, and it fixed that problem [19:41] bswartz: cool [19:48] nacc: where do I express the build dependencies other than in the .dsc file? [19:49] bswartz: debian/control in the source pacakge [19:49] aha! [19:49] bswartz: sorry :) [19:50] bswartz: right so you'd edit that file, updat the changelog and then rebuild the source pacakge (which will generate a new .dsc in the parent directory) [19:52] -clipman-plugin is currently in sync with Debian, it'd be faaaaar better to fix it there. [19:52] bswartz: and to be clear, note that you first need to fix anything in zesty and then follow https://wiki.ubuntu.com/StableReleaseUpdates for getting it into older releases, if applicable [19:53] Unit193: excellent point, i figured that might be the response in the bug [19:53] nacc: why did lp-attach launch lynx? [19:53] >_< [19:53] bswartz: hrm, not sure? maybe it tried to open the bug? if you piped to it, it doesn't do that, at least last i used it [19:54] bswartz: you can also manually take the debdiff and attach it to the bug [19:54] it's trying to do an openid transaction [19:54] bswartz: oh because you need to be auth'd to use it [19:54] oh well I can sort this out with a real web browser later [19:54] bswartz: sorry, forgot that bit (i'm always auth'd at home) [19:54] thanks or you help [19:56] Unit193: what is the process for debian? do they use launchpad too? [19:56] Generally file a bug and attach a debdiff. [19:56] Unit193: file a bug in launchpad? what project? [19:57] bswartz: you can use `reportbug -B debian` [19:57] bswartz: no, debian uses bugs.debian.org [19:58] k === alexisb is now known as alexisb-afk [20:42] I'm having a strange issue here. My zesty vm seems to be unable to ping or browser .org domains (for instance http://debian.org). Anyone know if this is a known issue or what the cause might be? My host (16.04) has no problems. === alexisb-afk is now known as alexisb [20:49] hjd: why do you mention .org domains? do .com domains work fine? [20:51] Yes, various .com domains, .no domains and launchpad.net all work. I first thought it was something with debian.org, but I tried some other random .org domains and they all keep failing for some reason [21:20] hjd: I'd recommend testing whether DNS is resolving okay and if not, use a different resolver [21:21] hjd: if DNS is resolving for those org domains then you have a routing or firewall problem === salem_ is now known as _salem === _salem is now known as salem_ [21:34] hjd: type 'host foo.org' on the commandline and see if that resolves [21:39] sladen: http://paste.ubuntu.com/23492450/ [21:40] I can ping the ip adresse just fine, but if I try to ping wikipedia.org I only get "Name or service not known" [21:42] hjd: is there a difference depending on if you 'ping 2620:0:862:ed1a::1' or if you 'ping 91.198.174.192' ? [21:42] sladen: you mean ping6 [21:44] bswartz: yes, the aim is to see whether it might be IPv4 vs. IPv6. The particular choice of tools is optional :) [21:44] IPv4 works, IPv6 Network is unreachable [21:45] hjd: if it turns out your problems are related to IPv6 being prefered over IPv4 then you can easily change the preference with this one-liner: [21:45] echo 'precedence ::ffff:0:0/96 100' | sudo tee -a /etc/gai.conf [21:56] bswartz: Hm... didn't seem to make a difference. [21:57] Anyway, it's getting late, but thanks for the help all :) === salem_ is now known as _salem