[00:06] <nacc> rbasak: well that's nice to see -- latest importer picked up the unapplied where it was, and is imported all the applied, so far successfully, so i can do the php7.0 merge
[00:12] <rbasak> Nice!
[15:31] <nacc> rbasak: ah i see i had to revert HEAD -- we need an indevelopment feature of git to allow us to skip known empty changes
[15:32] <nacc> rbasak: but that would end up skipping (potentially) changes that are now empty becuase they can be dropped
[15:32] <nacc> rbasak: which we don't want
[15:33] <rbasak> I see, OK.
[15:34] <nacc> rbasak: i put a comment in the comment with a link to a patch for git that might let us specify it more clearly
[15:34] <nacc> rbasak: already empty commits vs. commits that have become empty
[15:35] <nacc> rbasak: but not yet existent, so i reverted from `usd merge` and fixed the wiki
[16:05] <juliank> arges: I saw that you accepted apt 1.2.21 and 1.3.6 for xenial/yakkety, but not 1.4.1~17.04.1 for zesty. Any specific reason for that?
[16:06] <juliank> It's a bit odd to have the SRUs for older releases in -proposed without the one for the newer release
[16:31] <LocutusOfBorg> Noskcaj, FYI I'll sync dhelp as soon as it is picked up
[16:31] <LocutusOfBorg> after only 9 years seems that LP: #205308 is fixed
[16:31] <LocutusOfBorg> cjwatson, ^^ :)
[16:31] <xnox> juliank, we have more stuff to do for apt =)))))) we shall talk ;-) in a second.
[16:46] <juliank> xnox: More stuff to do sounds scary
[16:48] <juliank> My veggies are in the oven now, so I do have some time :)
[16:48] <xnox> juliank, hehehehe =) i had steamed veggies, quicker ;-)
[16:49] <xnox> juliank, so. we spoke to sysadmins and we have a huge spike at 6am..6:30am UTC from all the pre-xenial machines.
[16:49] <juliank> Right. That's expected
[16:50] <xnox> juliank, switching xenial+ machines to that window will decrease the baseload, and will spike the window even more. Which will take us over our bandwidth limit in the key datacentres.
[16:50] <xnox> juliank, but we had a brainstorm session and this is what we established.
[16:51] <xnox> juliank, currently we have 1 timer which triggers/does 3 things as a monolithic block: apt update; apt upgrade --download-only; apt upgrade.
[16:51] <xnox> the first two things, ideally should happen universally spread across the 24h window.
[16:51] <xnox> the end users (server/cloud sysadmins) care a lot about the precise time of upgrades application. And they do not care when the bandwidth is used up.
[16:51] <juliank> Well, yes, that's not really new :)
[16:52] <juliank> What about memory usage during update, though?
[16:52] <xnox> thus my current proposal is to decouple unattended upgrades application; from the update  & download-only
[16:52] <juliank> slangasek said his machine went really slow.
[16:52] <juliank> at the 1800 update due to memory spikes
[16:52] <slangasek> juliank: xnox speaks for me
[16:53] <xnox> juliank, i did not consider memory usage at all. And my expectations is that one should not run ubuntu if one cannot do apt update.
[16:53] <slangasek> juliank: fwiw I think it was ultimately the unattended-upgrades part (auto-installation of security updates) that broke me rather than the list updates
[16:53] <xnox> slangasek, plus i suspect slangasek has like a gazzilion of repositories enabled, unlike our core user base, which is actually hitting security pocket only.
[16:53] <slangasek> which I hadn't considered when I talked to you
[16:53] <juliank> OK
[16:54] <slangasek> juliank: also, fundamentally, 1) I've upgraded the RAM in my machine since then :P, and 2) I've been larted by our IS team ;-)
[16:54] <juliank> What is "larted"?
[16:54] <xnox> juliank, currently i'm writting up a back report with the decoupling plan, and will work on updating the units and/or configs; in apt and/or unattended upgrades. Such that the following behaviour happens for the long running instances:
[16:55] <xnox> run apt update smeared across 24h; if unattended-upgrades are enabled, execute --download-only straight after; and make unattanded-upgrades scheduled at the 6am..7am window.
[16:55] <xnox> on boot/resume: if we missed the cycle, execute on boot the whole lot, as enabled.
[16:55] <juliank> That's what I'd have seen as the optimal solution last week ...
[16:56] <xnox> juliank, this kind of means we are putting the sru on hold =/
[16:56] <xnox> juliank, if you want the other fixes in, i can redo the sru's without the timer change.
[16:56] <xnox> juliank, unless you really want to DDoS canonical datacentres and see an angry elmo =)
[16:56] <juliank> xnox: Nah
[16:58] <juliank> xnox: But then we basically just have to revert the change in apt, and set APT::Periodic::Unattended-Upgrade to 0 in unattended-upgrades, and add a timer for that
[16:58] <xnox> juliank, yes.
[16:58] <juliank> Although maybe we want 0:00 as time + random delay of 24 hours instead of two start + 12 hour delay
[16:58] <xnox> juliank, and the unattended-upgrade timer will be in the unattended-upgrade package.
[16:59] <juliank> Yep, sure
[16:59] <xnox> juliank, i think two times a day is good.
[16:59] <xnox> juliank, because on ubuntu it has other effects apart from unattaned upgrades.
[16:59] <xnox> there is MOTD updates; and other messenging that is triggered by apt update.
[17:00] <juliank> You gotta realize that there is a timestamp file in the apt script itself, so you could run it every minute, it only does something if 24 hours have passed since the last time
[17:00] <juliank> :D
[17:00] <xnox> slangasek, i am also affected!!!!!!! i've now realised why i have random alerts texted to me on telegram whenever my database/webserver are down. And it was all weird to me, as to why they are restarting, when i didn't touch it.
[17:01] <xnox> juliank, can we get rid of that, at least in ubuntu, and rely on the systemd timer alone?
[17:01] <xnox> (persistent systemd timer alone)
[17:01] <xnox> juliank, doesn't like anacron keep persistent timestamps like that as well?
[17:01] <juliank> Well, the script has multiple distinct timers
[17:01] <xnox> /o\
[17:01] <juliank> APT::Periodic::AutocleanInterval
[17:01] <juliank> APT::Periodic::CleanInterval
[17:02] <juliank> APT::Periodic::Update-Package-Lists
[17:02] <juliank> APT::Periodic::Download-Upgradeable-Packages
[17:02] <juliank> APT::Periodic::BackupArchiveInterval
[17:02] <juliank> all in the unit of days
[17:02] <xnox> juliank, so apt can be a cron replacement?! =)
[17:03] <juliank> Maybe it makes sense to split this into 5 systemd timers, but that's kind of weird
[17:04] <juliank> The two times a day should be enough to make it run often enough
[17:04] <xnox> juliank, not 5. Update-Package-Lists & Download-Upgradeable-Packages should be one timer triggered thing. And the rest could be the other.
[17:05] <xnox> because you do want to download things, straight after update, if enabled to do so.
[17:05] <juliank> Well, you say that.
[17:05] <xnox> (e.g. if unattended upgrades are enabled, or download-upgradable-packages is enabled)
[17:05] <xnox> i totally want to support download Update-Package-Lists only.
[17:08] <juliank> xnox: Now, I'd really like to have the same logic in Debian, I don't really want to have distinct Ubuntu branches again :)
[17:09] <juliank> So, I'd like keep the current setting of 6 to 7, and install a drop-in unit from the Ubuntu vendor directory.
[17:09] <xnox> juliank, are unattended upgrades enabled by default in debian?
[17:09] <juliank> Does that make sense?
[17:09] <juliank> xnox: I hope not.
[17:09] <xnox> juliank, drop-in: yes, that is fine, the ubuntu drop-in can go into /lib/systemd/system/apt-update-whatever-the-name-is.timer.d/ubuntu.conf
[17:09] <juliank> But we actually should have at least fixed the race conditions now, so that's something :)
[17:09] <xnox> [Timer]
[17:09] <xnox> OnCalendar=
[17:10] <xnox> OnCalendar=<ubuntu setting>
[17:10] <juliank> Yep
[17:11] <juliank> + Adjustments for RandomizedDelay
[17:14] <juliank> xnox: I also see u-u already has a service
[17:14] <juliank> So adding a timer should be easy
[17:15] <juliank> Ah, but that's just for shutdown
[17:17] <xnox> juliank, i think it would be more elaborate yes.
[17:17] <xnox> juliank, i was thinking to override apt settings when calling apt timers, and in ubuntu at least have something with -oAPT::DO-THIS-ONLY
[17:17] <xnox> and -oAPT::DO_THAT_ONLY
[17:18] <xnox> or some such, or more verbose scripts that determine what happens when.
[17:18] <xnox> i'm currently writting the plan up, and will be working on a proposal of patches to apt and/or unattended-upgrades packages
[17:20] <juliank> xnox: The script does not take -o options, that's very ... involved.
[17:21] <juliank> You could use config files, but I think it's pointless.
[17:22] <juliank> What you want as a default according to your description is APT::Periodic::Update-Package-Lists=1, APT::Periodic::Download-Upgradeable-Packages=1 and APT::Periodic::Unattended-Upgrade=0
[17:22] <slangasek> juliank: larted> from LART, luser attitude readjustment tool ;)
[17:22] <juliank> and then just run unattended-upgrades from the unattended-upgrades-daily service or whatever
[17:23] <juliank> Although, clean and autoclean would be weird in the apt job
[17:23] <juliank> Can we just split up the apt job in two?
[17:23] <xnox> juliank, clean and autoclean should be in the unattended-ugprade now?
[17:23] <xnox> juliank, we can do that too.
[17:23] <juliank> clean and autoclean run after unattended-upgrades
[17:24] <juliank> if configured
[17:24] <juliank> So if you'd move only unattended-upgrades out, you'd end up with update, download, clean if configured, and your files would be gone again
[17:25] <juliank> This way we can have apt-daily and apt-daily-dangerous
[17:25] <juliank> :D
[17:25] <juliank> (OK, maybe a better name for the second one)
[17:28] <juliank> I guess we can specify After betweens them, so *if* both are started at the same time, the upgrade&clean job runs later
[17:29] <xnox> juliank, yeah, we will need to do the after/before orderings right, for the onboot case.
[17:29] <xnox> juliank, so could you please checkout and comment on:
[17:29] <xnox> https://bugs.launchpad.net/ubuntu/artful/+source/unattended-upgrades/+bug/1686470
[17:30] <juliank> That sounds right
[17:31] <juliank> If we just split up the apt job in two, we can just keep the changes localized to apt though, and not change unattended-upgrades at all :D
[17:32] <xnox> yeah \o/ no need to do versioned breaks and depends
[17:33] <xnox> juliank, u-u service is for shutdown, i believe. not for on-boot stuff.
[17:35] <juliank> xnox: yeah
[17:36] <juliank> xnox: Another advantage of that scheme is that I can probably sell that the Debian release team :D
[17:37] <juliank> "Oh, we discovered that this might cause unwanted spikes in memory bandwidth if a lot of machines update at the same time, let's just do this all day, and special case upgrades"
[17:39] <xnox> juliank, yes.
[17:40] <xnox> juliank, "memory bandwidth" you mean "network bandwidth" right?
[17:40] <juliank> yes
[17:40] <juliank> odd mistake
[17:41] <xnox> juliank, and not can, but does take out a DC center we can see spikes to 10Gbps, vs the baseload of 4Gbps and some of our pipes are limited to 10Gbps
[17:42] <xnox> at which point we have protections to limit the bandwidth, such that #is can ssh into those boxes at least; and end users get timeouts doing apt update.
[17:44] <juliank> xnox: The diff is stupid, but https://github.com/Debian/apt/compare/master...julian-klode:lp1686470?expand=1 should be it on the script side
[17:44] <juliank> + install rules and a new timer and service
[17:45] <xnox> destructive-daily
[17:45]  * xnox giggles
[17:46] <juliank> xnox: Well, find a better name!
[17:48] <xnox> apt-download-updates, apt-install-upgrades
[17:48] <xnox> ... by definition installing things, breaks toys.
[17:48] <xnox> juliank, however we should probably not change the original timer name though =/ because $reasons
[17:52] <juliank> xnox: Yep. Maybe I should also keep the script together and just put the parts in an if/else, that's a bit easier to look at and build.
[17:53] <juliank> Then we can just pass upgrade as $1 to run upgrade, update to update, and if nothing passed it does all, or something like that
[17:53] <juliank> Has the benefit of being completely easy to review
[17:53] <juliank> just do a diff -w
[17:53] <juliank> :D
[18:00] <juliank> https://paste.ubuntu.com/24461931/
[18:06] <juliank> xnox: With the additional timer and service: https://paste.ubuntu.com/24461953/
[18:06] <juliank> Oh, also needs an adjustment to rules
[18:07] <juliank> that's a -w git, the unreadable version is at https://github.com/Debian/apt/compare/master...julian-klode:lp1686470?diff=split&expand=1&name=lp1686470
[18:09] <juliank> Exciting times!
[18:10] <juliank> I guess I'll just put this in a PPA quickly for zesty and artful to play with
[18:11] <juliank> But first, my own system
[18:44] <juliank> xnox: Stuff seems ready to go (built successfully, and I installed it :D), I could upload the branch to artful to give it a test drive.
[18:45] <xnox> juliank, artful would be nice; many of us run artful things.
[18:49] <juliank> xnox: One question, though: Does it make sense to have the u-u/clean stuff not depend on network-online, as I did, or should both depend on that?
[18:49] <juliank> I wonder if u-u works without network
[18:50] <xnox> juliank, u-u/clean should not need network, correct.
[18:50] <xnox> and i do not care about msft-fonts-package that does wget in postinst =)
[18:50] <juliank> It's theoretical anyway.
[18:51] <juliank> The network-online target is a boot thing
[18:51] <juliank> and apt-daily-upgrade has after=apt-daily which has after=network-online
[18:51] <juliank> (and wants=network-online)
[18:54] <xnox> juliank, possibly both, however the chain should be enough
[18:54] <juliank> yeah
[19:02] <juliank> xnox: Currently writing an email to deity@l.d.o and debian-release, do you want to be CCed on that?
[19:03] <juliank> (and anyone else)
[19:03] <juliank> Mostly collecting additional feedback
[19:08] <xnox> juliank, sure
[19:09] <juliank> xnox: which email address?
[19:11] <xnox> juliank, xnox@ anything - ubuntu.com, debian.org, spi-inc.org, surgut.co.uk......
[19:11] <xnox> as you wish
[19:11]  * xnox wonders surely everyones email addresses are predictable =)
[19:13] <juliank> xnox: Some people have preferences :)
[19:13] <juliank> Some people even commit with different emails when working vs non-working
[19:13] <xnox> some people have too much time on their hands =)
[19:14] <juliank> xnox: I do this for changelog entries, signing the ubuntu ones with juliank@ubuntu.com and the Debian ones with jak@debian.org
[19:15] <juliank> The commits all use my @debian.org, though
[19:15] <juliank> git should really allow me to set up per-branch emails
[19:15] <juliank> Author: jak@debian.org Committer: juliank@ubuntu.com
[19:16]  * juliank wishes jak@ubuntu.com were an alias for juliank@ubuntu.com
[19:17] <juliank> That account is not even used: https://launchpad.net/~jak
[19:17] <juliank> (and how did John Knottenbelt became "jak", where does the a come from ...)
[19:17] <Pici> middle name?
[19:18] <juliank> Probably from the email address?
[19:18] <juliank> anyway, that was just as unfortunate as jak being blocked on freenode and oftc
[19:18] <sarnold> juliank?
[19:19] <juliank> sarnold: yes?
[19:19] <sarnold> juliank: what do you mean 'blocked on oftc'?
[19:19] <juliank> well, registered by another user
[19:20] <juliank> Imagine if I had a single nick everywhere
[19:20] <juliank> sarnold: I almost bought jak.fun recently
[19:20]  * juliank would have been jak@jak.fun
[19:21] <sarnold> juliank: 'jak' is unused forever; if you want it, I'll drop it for you
[19:21] <juliank> I think it's a bit late to change nicks now :)
[19:21] <sarnold> juliank: 'jak' is unused on freenode for 36 weeks too, they may be willing to drop it here, too
[19:24] <juliank> sarnold: Maybe on OFTC, but I'm not sure if that does not confuse people who are both in Debian & Ubuntu :)
[19:24] <juliank> That said, highlighting me is not a problem - I react to both :D
[19:47] <juliank> OK, I think I can go relax now, and hope I did not break any artfuls
[20:48] <nacc> smoser: for orig tarball imports -- what should we do if debian (or ubuntu) switches, for the same upstream, from .gz to .bz2? we correctly don't see the bz2 in `pristine-tar list` but we can't import it, because the same upstream version has already been imported. this happened for src:acl between 2.2.51-1 and 2.2.51-2
[20:53] <nacc> infinity: slangasek: --^ if you have an opinion/advice on ideal behavior. Basically the importer we have no will use the 'first' orig tarball for a given upstream version as uploaded. But we don't verify if we see a second (different named) tarball for the same upstream that the contents match or anytyhing (currently)
[20:54] <jbicha> Ubuntu won't allow you to use two different tarballs for the same upstream version
[20:55] <nacc> jbicha: yes, but debian did
[20:55] <nacc> well "different" in that they have different names
[20:55] <infinity> jbicha: We sure will.
[20:56] <nacc> their contents might very well be the same
[20:56] <nacc> oh and actually, i found a case where what you said wasn't true, jbicha :)
[20:56] <nacc> i'd have to go look it up, but it does confuse LP a bit :)
[20:56] <infinity> jbicha: As long as the filename changes, you can change taballs all you want.  People often take advantage of that "feature" to use a cleaner/better upstream tarball by switching file compression.
[20:58] <infinity> nacc: So, I'm not sure how the importer should deal with this, but when people switch compression, they almost always do it intentionally to force a new tarball that *does* have different contents, so you certainly can't assume that it's still the same.
[20:58] <infinity> (I'm not saying this practice is sane or rational, but it's certainly not uncommon)
[21:01] <nacc> infinity: yeah, that's what i was thinking -- i'm not sure gbp-import-orig can handle this, which means i'll need to rethink that part
[21:03] <infinity> nacc: The fundamental problem here is that, while VCS workflows (and dpkg/Debian versioning Policy) make a big fuss about upstream version, and tracking an upstream series/branch, the archives (dak and soyuz) literally don't care.  upstream version is meaningless to archive software, all that matters is file consistency.  So, you can't upload foo_1.2.3.orig.tar.gz twice with different contents, but change the filename and all bets are off.
[21:04] <infinity> nacc: How you represent that eventuality back in the VCS world where "upstream version" actually means something, I'm not sure.  You might just need a different branch.  1.2.3^xz or something.
[21:04] <nacc> infinity: yep, understood -- i just want, if possible, to be able to reproduce a given build's files from our git tree.
[21:04] <nacc> infinity: yeah, i'm thinking we'd need to do something like that
[21:04] <infinity> (Bonus points for finding a char that's legal in git refs but illegal in dpkg versions)
[21:04] <nacc> infinity: i think we can also do something with the version tag
[21:04] <nacc> *upstream tag, rather
[21:04] <nacc> i'll need to play with it a bit to make everything happy
[21:04] <nacc> infinity: thanks for the feedback!
[22:55] <hloeung> juliank, xnox, slangasek: thanks for all your work on LP#1686470
[23:01] <juliank> hloeung: Let's see tomorrow if that actually works or if I messed up :)
[23:01] <hloeung> heh
[23:05] <juliank> We should build test cases for the daily script, actually
[23:05] <juliank> Well, unit tests
[23:06] <juliank> Just gotta mock various tools like apt-get and u-u
[23:07] <Unit193> I mock a lot of tools
[23:07] <sarnold> :)
[23:11] <Unit193> sarnold: I presume there's no update that you know of to 716535?
[23:12] <sarnold> Unit193: correct
[23:12] <sarnold> Unit193: or, rather, none that I've heard
[23:12] <sarnold> oh I see that's explicit in your question. sigh. :)
[23:13] <Unit193> Yep, you seem to "know everything" for the sec team, and easy to talk to.
[23:13]  * sarnold blushes
[23:14] <nacc> mdeslaur: just an fyi, i'll be sru'ing php7.0 7.0.18 shortly (hopefully today, or tmrw)
[23:18] <nacc> infinity: ah, it's actually quite a bit easier (for our case), we just need to make the tag names unique, so we can append the extension
[23:18] <nacc> infinity: that's the only part that fails for `gbp import-orig`
[23:23] <nacc> infinity: in any case, thank you again for the insight!
[23:38] <mdeslaur> nacc: ack, thanks