[01:04] does launchpad show ppa usage/download statistics anywhere? [01:08] actually, may be better to ask: is there a reason this would time-out? https://api.launchpad.net/1.0/~nginx/+archive/ubuntu/stable/+binarypub/40721078?ws.op=getDailyDownloadTotals&start_date=2013-10-23 (Error ID OOPS-debf0694dfb4065abd2d84ccc9668f58) [01:08] https://oops.canonical.com/?oopsid=OOPS-debf0694dfb4065abd2d84ccc9668f58 [01:08] (generated by a ppastats tool) [01:09] teward: That's a very long period. You'll get better results if you don't ask for almost two years of data at a time. [01:10] wgrant: FYI: I didn't generate the link? [01:10] nor do I have the capability to customize [01:10] teward: You'll want to tweak the code that you're using. [01:10] Why not? [01:10] http://wpitchoune.net/blog/ppastats/ [01:10] ^ that tool [01:10] Why can't you customise it? [01:11] I could customize it, but i'm not sure whether the release team *wants* that amount of data or not [01:11] only reason i'm using it is because the release team asked for ppa stats [01:11] not sure if giving them a smaller dataset would be frowned on by them [01:11] You can give them the entire dataset. [01:11] (see https://lists.ubuntu.com/archives/ubuntu-release/2015-July/003310.html) [01:11] You just can't request absolutely everything in one hit. [01:11] wgrant: then what do you suggest I do to the program and API reqs? [01:11] because I haven't dissected the tool's code [01:12] dear lord [01:12] it is an LP API client written in C [01:13] * teward yawns [01:13] wgrant: i'm tired, cut me some slack? [01:13] blame the holidays >.> [01:13] Oh, I'm just looking at the code, expecting it to be a 50-line Python script. [01:13] heheh [01:13] Instead it's thousands of lines of C... [01:14] wgrant: i think what the release team wants is nice graphics [01:14] this thing produces graphs apparently [01:19] teward: Anyway, as a general rule, if requesting two years of data times out, request less data at once. [01:19] wgrant: and without C coding knowledge (I have limited) I can't customize the tool. [01:19] That's not something I can help with. [01:19] indeed. [01:20] If it were Python, the code would be perhaps one-twentieth the size, and would be easy to adjust. [01:20] mhm [01:20] Python++ [01:20] C -- [01:21] Retrying it several times may work, but it may not. [01:21] it's tried 35 times so far *shrugs* [01:46] wait what [01:46] someone wrote non-kernel code to talk to an API in C?! [01:47] Yes, I thought I was drearming. [01:47] That 4000 LOC could probably be less than 200 lines of Python. [01:50] well, there's something charmingly perverse about that I guess. [01:50] no [01:50] there really isn't [01:50] No. [01:50] Nothing charming at all. [01:50] Perverse, sure :) [01:51] come on, it's a little amusing. [01:51] True. [01:51] if you find more buffer overflows in the world amusing === Nigel_ is now known as G === psivaa is now known as psivaa-afk [15:34] wgrant: lifeless: I guess you folks haven't heard of https://github.com/okws/okws :P [17:00] hey, any launchpadlib hackers around? [17:01] https://bugs.launchpad.net/launchpadlib/+bug/1471894 [17:01] Launchpad bug 1471894 in launchpadlib "_bad_oauth_token crashes on python3 (str vs bytes)" [Undecided,New] [17:13] zyga: doesn't crash here [17:13] python3-launchpadlib 1.10.3-1 [17:14] Ah, but I probably don't have an expired token [17:14] Yeah, should be trivial to fix, bonus points if your patch comes with a test [17:15] cjwatson: I have the code patched, patching existing tests not to crash with the patch in place [17:24] cjwatson: my current approach is to use b'...' constants instead of plain strings there [17:25] cjwatson: do you think that's the right way to do it? [17:25] Yes [17:25] cjwatson: (and patching tests to actually test with b'...' strings [17:25] That sounds fine [17:25] cjwatson: which is ok for python2 (no op) and gives us nice bytes in python3 [17:25] cjwatson: nice, thanks [17:25] Just check that if you temporarily shelve the code patch, the tests start failing in python3 [17:26] i.e. that the test changes constitute an effective test for this bug [17:28] cjwatson: sure [17:39] cjwatson: offtopic, I sent out a hello-world patch for tarmac but I didn't get any replies for it [17:40] cjwatson: I'm pretty booked lately but I will return to tarmac/git later this week [17:40] cjwatson: it would be good to have a person that can commit and can have a conversation with me [17:41] OK, I understand that, but it's not something I'm in a position to do much about [17:41] dobey: ^- can you help zyga? [17:41] cjwatson: sure, just something that flew past my mind while talking to you :) [17:41] cjwatson: iff there's nobody interested I could just be added to developer group [17:41] zyga: are you blocked on anything in LP for tarmac at the moment? (I know there are one or two things that are suboptimal) [17:42] cjwatson: no [17:42] cjwatson: I think it's okay now [17:42] https://bugs.launchpad.net/launchpad/+bug/1462449 is quite complicated, surprisingly [17:42] Launchpad bug 1462449 in Launchpad itself "branch_merge_proposal.reviewed_revid is '" if operated through anonymous API login" [High,Triaged] [17:42] cjwatson: those are not real blockers [17:42] I think that was your only recent unclosed bug [17:42] right, good [17:42] cjwatson: the most useful bits are already in [17:42] cjwatson: (thanks a lot for that!) [17:43] np [17:43] cjwatson: btw, if we're talking about git, I had inconsistent results lately [17:43] sent tarmac patch where? [17:43] cjwatson: sometimes I think I'm hitting a server that doesn't know my ssh key [17:43] dobey: merge request on lp:tarmac [17:43] dobey: it's not a real support for git, just a trivial rename to start the conversation with someone live [17:43] hmm [17:43] cjwatson: as repeated git push fix that [17:44] cjwatson: maybe related to ps4 outage [17:44] cjwatson: not sure, just letting you know [17:44] zyga: all the ps4 stuff should be out of DNS by now - check what DNS for git.launchpad.net gives you [17:44] git.launchpad.net has address 162.213.33.96 [17:44] git.launchpad.net has address 162.213.33.95 [17:44] right, those are ps4.5 [17:45] zyga: i'll take a look, but i suspect tarmac probably won't be able to support git, in its current design [17:45] dobey: yeah, I'm willing to do all the work to change that [17:45] dobey: I wanted to talk to someone about a plan how to approach that [17:45] dobey: and if that's something that would be merged [17:45] zyga: does git have something remotely anything like bzrlib? [17:45] tarmac really doesn't look all that far off supporting git [17:45] dobey: doing that in dry run mode and getting a nack later would be pretty costly for me:/ [17:46] dobey: there's pygit2, but honestly shelling out to plain git should perform fine [17:46] cjwatson: no result is impossible with an appropriate patch ;> [17:46] cjwatson: rm -rf; git add; profit ;-) [17:46] cjwatson: yeah, shelling out works great [17:46] git has nowhere near the startup time cost of bzr [17:46] dobey: git has many implementations [17:46] well, it doesn't have to start python and load all the plug-ins at init, sure [17:47] dobey: there's libgit2? from github folks (C) [17:47] pygit2 is based on libgit2 [17:47] dobey: but I think it's not useful to work at that level [17:47] but it also isn't in Ubuntu [17:47] dobey: as git CLI is just fast and everything is there [17:47] dobey: I've taken specific care to make the LP webservice interface reasonably compatible for bzr/git MPs, with reference to what tarmac is using [17:47] dobey: it's certainly not the bottleneck in the prototype I wrote [17:47] zyga: it's not about how fast the CLI is or isn't really. shelling out makes for not-so-nice code [17:48] cjwatson: sure, and launchpadlib should handle that part just fine; it's the local stuff i'm really worried about in that respect [17:48] dobey: I also think that but only to some degree, the prototype that shells out supports git and bzr and is 300 lines long [17:48] it would perhaps be better to use pygit2, but honestly, it doesn't seem reasonable to block on that [17:48] cjwatson: https://bugs.launchpad.net/launchpadlib/+bug/1471894/comments/1 [17:48] Launchpad bug 1471894 in launchpadlib "_bad_oauth_token crashes on python3 (str vs bytes)" [Undecided,New] [17:49] cjwatson: does that look reasonable? [17:49] zyga: I think so, but would rather have an MP :) [17:49] dobey: the problem with many apis is that we are all used to cli and sometimes it's not easy to understand which apis a given cli command maps to exactly, I think that applies to bzr and git equally well [17:49] cjwatson: sure, that's step two :) [17:51] python-pygit2 is actually in wily, so it would be possible to write against that [17:51] I just worry that it's an extra barrier for people looking to upgrade existing tarmac instances to support git MPs [17:51] cjwatson: I agree [17:51] cjwatson: I target LTS codebases typically [17:51] and I'd like that to be as low-friction as possible to make it easier for people to convert over [17:51] cjwatson: that's very good for adoption [17:52] tarmac isn't in the archive [17:52] so whether one of its dependencies is in the archive or not shouldn't really be a blocker, i don't think [17:52] we're using pygit2 in the LP git backend implementation, but we can also deal with getting our own dependencies excruciatingly right [17:52] I'm less sure about a bazillion little tarmac deployments [17:53] especially because libgit2/pygit2's API compatibility, well [17:53] i'm sure tarmac hasn't got so many users that it would be problematic :) [17:54] zyga: well, if dobey feels this is required, then pygit2 isn't that hard to use. we have reasonable experience with it in the LP dev team now [17:55] and there is a backport in ppa:launchpad/ubuntu/ppa [17:55] dobey: I think that this is actually a bad way to approach this, I think tarmac is just x10 larger than it has to be, has pointless deps and is tied to bzrlib+python2 [17:55] mostly i'd like to know what "the plan" would be, to get git support in, even if it actually did nothing at the moment in the case of git [17:55] dobey: but I'm very much open to discussion, I just come with opinions and, hopefully, arguments [17:55] what is a bad way to approach what? [17:56] dobey: using python-* for each VCS [17:56] dobey: as this requires working bindings and blocks python3 transition [17:56] dobey: shelling out is far easier if somewhat less technically pure in some way [17:56] (pygit2 has python3 support FYI) [17:56] cjwatson: and bzr? [17:56] cjwatson: run one installation with python2 for bzr, with python3 for git? [17:56] sure, I know that, but we're talking about git [17:56] cjwatson: that's a bit bad [17:57] it only blocks py3 for bzr support, and bzrlib is never going to support py3 afaik [17:57] cjwatson: sure but do we remove bzr to add git? [17:57] cjwatson: or require different runtime to use git? [17:57] no, please stop being disingenuous [17:57] it's not bad [17:57] no, we don't remove bzr support [17:57] my point is simply that the python3 point is not relevant for pygit2 [17:57] i think the python3 point is irrelevant anyway [17:57] it's a non-argument [17:57] dobey: why? so you want tarmac to be python2 forever/ [17:58] zyga: i don't care if it's python2 or 3; it's quite easy to support both until such a time bzr is not used by anyone any more who uses tarmac [17:58] * zyga pushes the fix to launchpadlib [17:59] i'm not going to strip tarmac down to be a shell script written in python, just to get rid of py2 support [17:59] dobey: so what kind of approach would you like to see [18:00] dobey: initially I did try to de-couple tarmac from bzrlib (to run on python3) and shell-out to bits for each vcs, but if you don't want to see that then another option must be available [18:00] zyga: i don't know exactly, but if there is to be support for multiple VCS types, then there obviously needs to be some sort of abstraction layer somewhere [18:01] tarmac.branch seems like about the right point for the abstraction layer [18:01] virtually all the operations there, with the exception of bug/branch links, have analogues in git [18:01] cjwatson: yes, I agree, to some extent, there's also some config changes that need to happen, tarmac ties entires there to branches directly and a new scheme would be needed for git [18:01] zyga: we can't shell out to bzr for all features in tarmac, so shelling is simply not an option for bzr at least [18:01] i have no idea whether all those features will even be supportable with git at all though [18:02] dobey: what features are those? [18:02] dobey: the only one I could find that's not (at least at present) is bug/branch links [18:02] tarmac.branch.Branch.fixed_bugs [18:02] dobey: from the top of my head I can think of the checkout-lock-held semantics [18:02] cjwatson: we also add custom metadata [18:02] dobey: hm, where's that? [18:03] oh, revprops['reviews']? [18:03] cjwatson: i'd have to go search the code, but we add revision propes for the reviewers, yeah [18:03] yeah that's it [18:03] dobey: what's the use of that? [18:03] The usual git idiom is to stuff that into the commit message :) [18:03] cjwatson: https://code.launchpad.net/~zyga/launchpadlib/fix-1471894/+merge/263950 [18:03] Which is kind of anathema to people used to bzrlib style, but meh [18:03] yeah, CI train does as well [18:04] and it makes commit messages quite ugly :( [18:04] Perhaps it would be OK to disable that for git, pending discussion? [18:05] cjwatson: well, since there's no way to put that (structured) in git then I think it's hard to make that an option, same as bug associations [18:05] It doesn't have to be structured, since git people don't expect it to be structure [18:05] well, there's a plug-in in tarmac which adds the info to the commit message too i think [18:05] d [18:05] dobey: what do you think about the checkout mode of operation [18:06] maybe it would be ok to drop that feature (we don't have a bzr plug-in to be able to pull it out on cli anyway) [18:06] dobey: which cannot be replicated with git [18:07] you could just have the commit implementation be commit + push [18:07] cjwatson: yes, I have that, it's just that the semantics is a little bit different due to the lock logic [18:07] zyga: i'm not sure what that means exactly. i think the best start would be abstracting out the bzr branch bits another level (as you've already started), and rewriting all the tests to not use the bzrlib testing stuff, but instead do abstract mocking or such [18:08] dobey: he means the bit where tarmac requires a checkout / bound branch so that it doesn't have to worry about lock failures, pushing, etc. [18:08] zyga: yeah, maybe ok as long as tarmac will retry the merge the next time it runs? [18:08] cjwatson: yeah i got that now. i was confused beause "checkout" in git is something else entirely :) [18:09] you just have to make sure to clean up on a push failure [18:09] cjwatson: mmm, yeah, it's just another failure that can happen [18:09] cjwatson: good point [18:09] which is easy if you do all the work in a temp branch [18:10] i'd also started some work to add signature verification to tarmac, but i haven't had any time to work on tarmac in quite a while :-/ [18:10] dobey: signature verification? [18:10] i don't know how signatures work in git [18:10] dobey: signed commits? [18:10] zyga: yes, gpg signed commits [18:10] dobey: ah, I think in git you more like gpg sign tags instead [18:10] dobey: though I haven't used that much myself [18:10] you can sign commits too, it's a topic of debate [18:11] I tend to just sign tags [18:11] zyga: thanks for the lplib branch, I'm EOD now but will remind myself where my test environment is in the morning [18:11] cjwatson: thanks! [18:12] cjwatson: I'll publish it in my ppa so my colleague can hack on that in a few hours when he wakes up [18:12] cjwatson: when you upload a new version to ubuntu I'll backport that as well [18:35] cjwatson: (one more) https://code.launchpad.net/~zyga/launchpadlib/fix-1471927/+merge/263953 === keithzg_vacation is now known as keithzg [20:02] hi; FYI an email from launchpad.net is seen in leaked NSA slides [20:02] large 43.6MB PDF unfortunately https://s3.amazonaws.com/s3.documentcloud.org/documents/2116130/intro-to-xks-appids-and-fingerprints.pdf [20:03] on page 24 of 60, a "Launchpad OpenPGP Key Confirmation" email [20:03] I wonder, were those emails sent to Ubuntu developers? or regular users too? [20:05] StevenC99: i'm not going to look deeper into the document given the top line says TOP SECRET [20:05] and i'm instead going to question where the hell you got the document from [20:06] sorry, should have perhaps linked to the news article instead [20:06] https://firstlook.org/theintercept/2015/07/02/look-under-hood-xkeyscore/ [20:06] teward: obviously it's declassified or something [20:06] dobey: 'leaked nsa slides' [20:06] i'm questioning that given the 'leaked nsa slides' part [20:06] StevenC99: the e-mail is the same type of e-mail sent to anyone to confirm adding a gpg key to launchpad, yes [20:06] (and when a new PGP key is added to Launchpad, LP emails the owner of the LP account an encrypted message they have to decrypt with their own private key corresponding to the PGP public key it was encrypted to, in order to provide the verification link to add it to the account) [20:07] it's the same type of email, yes. [20:07] StevenC99: having it used as an example of a pgp-encrypted messages doesn't mean much [20:07] ^ that [20:07] thanks; so this could be any Ubuntu user, not necessarily a developer [20:08] you should be more worried about all the e-mails you get from various services, in clear text, that have personal information and passwords in them [20:08] this was 2008; we didn't know better at the time [20:08] unless you can read under the redacted e-mail [20:08] eh, some of us knew better :) [20:09] i think this email was rather intercepted at the recipient's end, when they read it via plain HTTP webmail (mail/webmail/outblaze) [20:09] or who knows, maybe they did use HTTPS but it was broken somehow [20:10] if launchpad.net do happen to have email logs from Wed, 31 Dec 2008 10:04:16 -0000, they could maybe identify and notify this person [20:10] huh? [20:11] if launchpad.net has logs of who they send mail to at that specific time, they could figure out whose email this was [20:11] sent* [20:11] no i mean the interception part [20:11] oh okay, well... launchpad.net emailed somebody [20:11] that person then used some HTTP webmail client to read it [20:11] the nsa has been sucking up data off the wire for decades [20:11] and that's when I think NSA intercepted the email [20:12] i doubt it. it was probably intercepted when it went across the SMTP [20:14] i think it's a nice, chilling example though [20:17] come to think of it, there's a scerenshot of what looks like a webmail interface at the top [20:17] so surely, the person whose email is shown here, was the author of the NSA slide [20:18] well, not surely, but possible [20:19] i think you are making way too many assumptions about it :) [20:21] it would be cool to find out. launchpad.net email logs seem like the only way [20:21] unfortunately the first line of base64 is obscured, so i can't extract the GPG keyid from it [20:34] lfaraone: C++ concerns me much less