[00:06] rbasak: well that's nice to see -- latest importer picked up the unapplied where it was, and is imported all the applied, so far successfully, so i can do the php7.0 merge [00:12] Nice! === Spydar007 is now known as Guest92796 === cpaelzer_ is now known as cpaelzer === JanC is now known as Guest5365 === JanC_ is now known as JanC === klebers_ is now known as klebers === _salem is now known as salem_ [15:31] rbasak: ah i see i had to revert HEAD -- we need an indevelopment feature of git to allow us to skip known empty changes [15:32] rbasak: but that would end up skipping (potentially) changes that are now empty becuase they can be dropped [15:32] rbasak: which we don't want [15:33] I see, OK. [15:34] rbasak: i put a comment in the comment with a link to a patch for git that might let us specify it more clearly [15:34] rbasak: already empty commits vs. commits that have become empty [15:35] rbasak: but not yet existent, so i reverted from `usd merge` and fixed the wiki === salem_ is now known as _salem [16:05] arges: I saw that you accepted apt 1.2.21 and 1.3.6 for xenial/yakkety, but not 1.4.1~17.04.1 for zesty. Any specific reason for that? [16:06] It's a bit odd to have the SRUs for older releases in -proposed without the one for the newer release [16:31] Noskcaj, FYI I'll sync dhelp as soon as it is picked up [16:31] after only 9 years seems that LP: #205308 is fixed [16:31] Launchpad bug 205308 in dhelp (Ubuntu) "dhelp postinst error on gutsy->hardy upgrade" [High,Fix released] https://launchpad.net/bugs/205308 [16:31] cjwatson, ^^ :) [16:31] juliank, we have more stuff to do for apt =)))))) we shall talk ;-) in a second. [16:46] xnox: More stuff to do sounds scary [16:48] My veggies are in the oven now, so I do have some time :) [16:48] juliank, hehehehe =) i had steamed veggies, quicker ;-) [16:49] juliank, so. we spoke to sysadmins and we have a huge spike at 6am..6:30am UTC from all the pre-xenial machines. [16:49] Right. That's expected [16:50] juliank, switching xenial+ machines to that window will decrease the baseload, and will spike the window even more. Which will take us over our bandwidth limit in the key datacentres. [16:50] juliank, but we had a brainstorm session and this is what we established. [16:51] juliank, currently we have 1 timer which triggers/does 3 things as a monolithic block: apt update; apt upgrade --download-only; apt upgrade. [16:51] the first two things, ideally should happen universally spread across the 24h window. [16:51] the end users (server/cloud sysadmins) care a lot about the precise time of upgrades application. And they do not care when the bandwidth is used up. [16:51] Well, yes, that's not really new :) [16:52] What about memory usage during update, though? [16:52] thus my current proposal is to decouple unattended upgrades application; from the update & download-only [16:52] slangasek said his machine went really slow. [16:52] at the 1800 update due to memory spikes [16:52] juliank: xnox speaks for me [16:53] juliank, i did not consider memory usage at all. And my expectations is that one should not run ubuntu if one cannot do apt update. [16:53] juliank: fwiw I think it was ultimately the unattended-upgrades part (auto-installation of security updates) that broke me rather than the list updates [16:53] slangasek, plus i suspect slangasek has like a gazzilion of repositories enabled, unlike our core user base, which is actually hitting security pocket only. [16:53] which I hadn't considered when I talked to you [16:53] OK [16:54] juliank: also, fundamentally, 1) I've upgraded the RAM in my machine since then :P, and 2) I've been larted by our IS team ;-) [16:54] What is "larted"? [16:54] juliank, currently i'm writting up a back report with the decoupling plan, and will work on updating the units and/or configs; in apt and/or unattended upgrades. Such that the following behaviour happens for the long running instances: [16:55] run apt update smeared across 24h; if unattended-upgrades are enabled, execute --download-only straight after; and make unattanded-upgrades scheduled at the 6am..7am window. [16:55] on boot/resume: if we missed the cycle, execute on boot the whole lot, as enabled. [16:55] That's what I'd have seen as the optimal solution last week ... [16:56] juliank, this kind of means we are putting the sru on hold =/ [16:56] juliank, if you want the other fixes in, i can redo the sru's without the timer change. [16:56] juliank, unless you really want to DDoS canonical datacentres and see an angry elmo =) [16:56] xnox: Nah [16:58] xnox: But then we basically just have to revert the change in apt, and set APT::Periodic::Unattended-Upgrade to 0 in unattended-upgrades, and add a timer for that [16:58] juliank, yes. [16:58] Although maybe we want 0:00 as time + random delay of 24 hours instead of two start + 12 hour delay [16:58] juliank, and the unattended-upgrade timer will be in the unattended-upgrade package. [16:59] Yep, sure [16:59] juliank, i think two times a day is good. [16:59] juliank, because on ubuntu it has other effects apart from unattaned upgrades. [16:59] there is MOTD updates; and other messenging that is triggered by apt update. [17:00] You gotta realize that there is a timestamp file in the apt script itself, so you could run it every minute, it only does something if 24 hours have passed since the last time [17:00] :D [17:00] slangasek, i am also affected!!!!!!! i've now realised why i have random alerts texted to me on telegram whenever my database/webserver are down. And it was all weird to me, as to why they are restarting, when i didn't touch it. [17:01] juliank, can we get rid of that, at least in ubuntu, and rely on the systemd timer alone? [17:01] (persistent systemd timer alone) [17:01] juliank, doesn't like anacron keep persistent timestamps like that as well? [17:01] Well, the script has multiple distinct timers [17:01] /o\ [17:01] APT::Periodic::AutocleanInterval [17:01] APT::Periodic::CleanInterval [17:02] APT::Periodic::Update-Package-Lists [17:02] APT::Periodic::Download-Upgradeable-Packages [17:02] APT::Periodic::BackupArchiveInterval [17:02] all in the unit of days [17:02] juliank, so apt can be a cron replacement?! =) [17:03] Maybe it makes sense to split this into 5 systemd timers, but that's kind of weird [17:04] The two times a day should be enough to make it run often enough [17:04] juliank, not 5. Update-Package-Lists & Download-Upgradeable-Packages should be one timer triggered thing. And the rest could be the other. [17:05] because you do want to download things, straight after update, if enabled to do so. [17:05] Well, you say that. [17:05] (e.g. if unattended upgrades are enabled, or download-upgradable-packages is enabled) [17:05] i totally want to support download Update-Package-Lists only. [17:08] xnox: Now, I'd really like to have the same logic in Debian, I don't really want to have distinct Ubuntu branches again :) [17:09] So, I'd like keep the current setting of 6 to 7, and install a drop-in unit from the Ubuntu vendor directory. [17:09] juliank, are unattended upgrades enabled by default in debian? [17:09] Does that make sense? [17:09] xnox: I hope not. [17:09] juliank, drop-in: yes, that is fine, the ubuntu drop-in can go into /lib/systemd/system/apt-update-whatever-the-name-is.timer.d/ubuntu.conf [17:09] But we actually should have at least fixed the race conditions now, so that's something :) [17:09] [Timer] [17:09] OnCalendar= [17:10] OnCalendar= [17:10] Yep [17:11] + Adjustments for RandomizedDelay [17:14] xnox: I also see u-u already has a service [17:14] So adding a timer should be easy [17:15] Ah, but that's just for shutdown [17:17] juliank, i think it would be more elaborate yes. [17:17] juliank, i was thinking to override apt settings when calling apt timers, and in ubuntu at least have something with -oAPT::DO-THIS-ONLY [17:17] and -oAPT::DO_THAT_ONLY [17:18] or some such, or more verbose scripts that determine what happens when. [17:18] i'm currently writting the plan up, and will be working on a proposal of patches to apt and/or unattended-upgrades packages [17:20] xnox: The script does not take -o options, that's very ... involved. [17:21] You could use config files, but I think it's pointless. [17:22] What you want as a default according to your description is APT::Periodic::Update-Package-Lists=1, APT::Periodic::Download-Upgradeable-Packages=1 and APT::Periodic::Unattended-Upgrade=0 [17:22] juliank: larted> from LART, luser attitude readjustment tool ;) [17:22] and then just run unattended-upgrades from the unattended-upgrades-daily service or whatever [17:23] Although, clean and autoclean would be weird in the apt job [17:23] Can we just split up the apt job in two? [17:23] juliank, clean and autoclean should be in the unattended-ugprade now? [17:23] juliank, we can do that too. [17:23] clean and autoclean run after unattended-upgrades [17:24] if configured [17:24] So if you'd move only unattended-upgrades out, you'd end up with update, download, clean if configured, and your files would be gone again [17:25] This way we can have apt-daily and apt-daily-dangerous [17:25] :D [17:25] (OK, maybe a better name for the second one) [17:28] I guess we can specify After betweens them, so *if* both are started at the same time, the upgrade&clean job runs later [17:29] juliank, yeah, we will need to do the after/before orderings right, for the onboot case. [17:29] juliank, so could you please checkout and comment on: [17:29] https://bugs.launchpad.net/ubuntu/artful/+source/unattended-upgrades/+bug/1686470 [17:29] Launchpad bug 1686470 in unattended-upgrades (Ubuntu Artful) "Apt updates that are uniformally spread across all timezones, with predictable application windows" [High,Triaged] [17:30] That sounds right [17:31] If we just split up the apt job in two, we can just keep the changes localized to apt though, and not change unattended-upgrades at all :D [17:32] yeah \o/ no need to do versioned breaks and depends [17:33] juliank, u-u service is for shutdown, i believe. not for on-boot stuff. [17:35] xnox: yeah [17:36] xnox: Another advantage of that scheme is that I can probably sell that the Debian release team :D [17:37] "Oh, we discovered that this might cause unwanted spikes in memory bandwidth if a lot of machines update at the same time, let's just do this all day, and special case upgrades" [17:39] juliank, yes. [17:40] juliank, "memory bandwidth" you mean "network bandwidth" right? [17:40] yes [17:40] odd mistake [17:41] juliank, and not can, but does take out a DC center we can see spikes to 10Gbps, vs the baseload of 4Gbps and some of our pipes are limited to 10Gbps [17:42] at which point we have protections to limit the bandwidth, such that #is can ssh into those boxes at least; and end users get timeouts doing apt update. [17:44] xnox: The diff is stupid, but https://github.com/Debian/apt/compare/master...julian-klode:lp1686470?expand=1 should be it on the script side [17:44] + install rules and a new timer and service [17:45] destructive-daily [17:45] * xnox giggles [17:46] xnox: Well, find a better name! [17:48] apt-download-updates, apt-install-upgrades [17:48] ... by definition installing things, breaks toys. [17:48] juliank, however we should probably not change the original timer name though =/ because $reasons [17:52] xnox: Yep. Maybe I should also keep the script together and just put the parts in an if/else, that's a bit easier to look at and build. [17:53] Then we can just pass upgrade as $1 to run upgrade, update to update, and if nothing passed it does all, or something like that [17:53] Has the benefit of being completely easy to review [17:53] just do a diff -w [17:53] :D [18:00] https://paste.ubuntu.com/24461931/ [18:06] xnox: With the additional timer and service: https://paste.ubuntu.com/24461953/ [18:06] Oh, also needs an adjustment to rules [18:07] that's a -w git, the unreadable version is at https://github.com/Debian/apt/compare/master...julian-klode:lp1686470?diff=split&expand=1&name=lp1686470 [18:09] Exciting times! [18:10] I guess I'll just put this in a PPA quickly for zesty and artful to play with [18:11] But first, my own system [18:44] xnox: Stuff seems ready to go (built successfully, and I installed it :D), I could upload the branch to artful to give it a test drive. [18:45] juliank, artful would be nice; many of us run artful things. [18:49] xnox: One question, though: Does it make sense to have the u-u/clean stuff not depend on network-online, as I did, or should both depend on that? [18:49] I wonder if u-u works without network [18:50] juliank, u-u/clean should not need network, correct. [18:50] and i do not care about msft-fonts-package that does wget in postinst =) [18:50] It's theoretical anyway. [18:51] The network-online target is a boot thing [18:51] and apt-daily-upgrade has after=apt-daily which has after=network-online [18:51] (and wants=network-online) [18:54] juliank, possibly both, however the chain should be enough [18:54] yeah [19:02] xnox: Currently writing an email to deity@l.d.o and debian-release, do you want to be CCed on that? [19:03] (and anyone else) [19:03] Mostly collecting additional feedback [19:08] juliank, sure [19:09] xnox: which email address? [19:11] juliank, xnox@ anything - ubuntu.com, debian.org, spi-inc.org, surgut.co.uk...... [19:11] as you wish [19:11] * xnox wonders surely everyones email addresses are predictable =) [19:13] xnox: Some people have preferences :) [19:13] Some people even commit with different emails when working vs non-working [19:13] some people have too much time on their hands =) [19:14] xnox: I do this for changelog entries, signing the ubuntu ones with juliank@ubuntu.com and the Debian ones with jak@debian.org [19:15] The commits all use my @debian.org, though [19:15] git should really allow me to set up per-branch emails [19:15] Author: jak@debian.org Committer: juliank@ubuntu.com [19:16] * juliank wishes jak@ubuntu.com were an alias for juliank@ubuntu.com [19:17] That account is not even used: https://launchpad.net/~jak [19:17] (and how did John Knottenbelt became "jak", where does the a come from ...) [19:17] middle name? [19:18] Probably from the email address? [19:18] anyway, that was just as unfortunate as jak being blocked on freenode and oftc [19:18] juliank? [19:19] sarnold: yes? [19:19] juliank: what do you mean 'blocked on oftc'? [19:19] well, registered by another user [19:20] Imagine if I had a single nick everywhere [19:20] sarnold: I almost bought jak.fun recently [19:20] * juliank would have been jak@jak.fun [19:21] juliank: 'jak' is unused forever; if you want it, I'll drop it for you [19:21] I think it's a bit late to change nicks now :) [19:21] juliank: 'jak' is unused on freenode for 36 weeks too, they may be willing to drop it here, too [19:24] sarnold: Maybe on OFTC, but I'm not sure if that does not confuse people who are both in Debian & Ubuntu :) [19:24] That said, highlighting me is not a problem - I react to both :D [19:47] OK, I think I can go relax now, and hope I did not break any artfuls [20:48] smoser: for orig tarball imports -- what should we do if debian (or ubuntu) switches, for the same upstream, from .gz to .bz2? we correctly don't see the bz2 in `pristine-tar list` but we can't import it, because the same upstream version has already been imported. this happened for src:acl between 2.2.51-1 and 2.2.51-2 [20:53] infinity: slangasek: --^ if you have an opinion/advice on ideal behavior. Basically the importer we have no will use the 'first' orig tarball for a given upstream version as uploaded. But we don't verify if we see a second (different named) tarball for the same upstream that the contents match or anytyhing (currently) [20:54] Ubuntu won't allow you to use two different tarballs for the same upstream version [20:55] jbicha: yes, but debian did [20:55] well "different" in that they have different names [20:55] jbicha: We sure will. [20:56] their contents might very well be the same [20:56] oh and actually, i found a case where what you said wasn't true, jbicha :) [20:56] i'd have to go look it up, but it does confuse LP a bit :) [20:56] jbicha: As long as the filename changes, you can change taballs all you want. People often take advantage of that "feature" to use a cleaner/better upstream tarball by switching file compression. [20:58] nacc: So, I'm not sure how the importer should deal with this, but when people switch compression, they almost always do it intentionally to force a new tarball that *does* have different contents, so you certainly can't assume that it's still the same. [20:58] (I'm not saying this practice is sane or rational, but it's certainly not uncommon) [21:01] infinity: yeah, that's what i was thinking -- i'm not sure gbp-import-orig can handle this, which means i'll need to rethink that part [21:03] nacc: The fundamental problem here is that, while VCS workflows (and dpkg/Debian versioning Policy) make a big fuss about upstream version, and tracking an upstream series/branch, the archives (dak and soyuz) literally don't care. upstream version is meaningless to archive software, all that matters is file consistency. So, you can't upload foo_1.2.3.orig.tar.gz twice with different contents, but change the filename and all bets are off. [21:04] nacc: How you represent that eventuality back in the VCS world where "upstream version" actually means something, I'm not sure. You might just need a different branch. 1.2.3^xz or something. [21:04] infinity: yep, understood -- i just want, if possible, to be able to reproduce a given build's files from our git tree. [21:04] infinity: yeah, i'm thinking we'd need to do something like that [21:04] (Bonus points for finding a char that's legal in git refs but illegal in dpkg versions) [21:04] infinity: i think we can also do something with the version tag [21:04] *upstream tag, rather [21:04] i'll need to play with it a bit to make everything happy [21:04] infinity: thanks for the feedback! [22:55] juliank, xnox, slangasek: thanks for all your work on LP#1686470 [23:01] hloeung: Let's see tomorrow if that actually works or if I messed up :) [23:01] heh [23:05] We should build test cases for the daily script, actually [23:05] Well, unit tests [23:06] Just gotta mock various tools like apt-get and u-u [23:07] I mock a lot of tools [23:07] :) [23:11] sarnold: I presume there's no update that you know of to 716535? [23:12] Unit193: correct [23:12] Unit193: or, rather, none that I've heard [23:12] oh I see that's explicit in your question. sigh. :) [23:13] Yep, you seem to "know everything" for the sec team, and easy to talk to. [23:13] * sarnold blushes [23:14] mdeslaur: just an fyi, i'll be sru'ing php7.0 7.0.18 shortly (hopefully today, or tmrw) [23:18] infinity: ah, it's actually quite a bit easier (for our case), we just need to make the tag names unique, so we can append the extension [23:18] infinity: that's the only part that fails for `gbp import-orig` [23:23] infinity: in any case, thank you again for the insight! [23:38] nacc: ack, thanks