[06:53] <davidcalle> Morning o/
[06:58] <dholbach> davidcalle, yo yo yo
[06:59] <dholbach> I didn't play around with staging too much yet
[06:59] <dholbach> and haven't figured out the hero tour
[06:59] <dholbach> I'm still going through the examples and making sure that stuff works or almost works with the new snapd/series 16
[07:02] <davidcalle> Yep, btw, youtube-dl works for me
[07:03] <dholbach> built from the newest trunk with the newest snapd and snapcraft?
[07:03] <davidcalle> A fresh build worked yesterday morning, but let me update and try again
[07:03] <davidcalle> *evening
[07:04] <dholbach> thanks
[07:05] <dholbach> oh, and install squid-deb-proxy{,-client} before :)
[07:06] <davidcalle> Oh no wait, I haven't tried a fresh build actually, I just took dpm's word
[07:07] <dholbach> big mistake :-P
[07:07] <davidcalle> dholbach: how do you start webdm? I used to do it with snappy service webdm start
[07:07] <dholbach> I have no idea
[07:30] <davidcalle> dholbach: do you still have ubuntu-core-config installed?
[07:30] <dholbach> I managed to fix a number of issues in lp:snappy-playpen, unfortunately this didn't give us more working snaps
[07:30] <dholbach> davidcalle, yes... should I delete it? :)
[07:30] <davidcalle> No, just checking if I should :)
[07:31] <dholbach> You should not need to
[07:31] <dholbach>  install this package manually.
[07:31] <dholbach> Not sure what that's supposed to mean... :)
[07:33] <davidcalle> :)
[08:04] <davidcalle> dholbach: youtube-dl works perfectly for me :)
[08:05] <dholbach> ok... maybe you can follow up on the bug report then?
[08:12] <davidcalle> ok
[09:25] <dholbach> davidcalle, the caching on staging (or whatever it is) is driving me nuts
[09:27] <davidcalle> dholbach, I feel your pain!
[09:29] <davidcalle> dholbach, will you mind if I wipe the staging db later today? (around 15)
[09:30] <dholbach> no, not at all
[09:30] <dholbach> I should be done with my tests before
[09:30] <davidcalle> ok
[09:30] <dholbach> during the import:
[09:30] <dholbach> 502 Bad Gateway
[09:30] <dholbach> The server returned an invalid or incomplete response.
[09:30] <dholbach> do we have a way to access logs?
[09:30] <dholbach> or do we need to ask IS/webops?
[09:30] <dholbach> it's a bit hard to debug without logs
[09:31] <davidcalle> we have, I'll send them to you in 2 min
[09:32] <dholbach> <3
[09:39] <davidcalle> dholbach: any specific log file?
[09:39] <davidcalle> import_md.log is empty
[09:39] <dholbach> sure, that's probably only used for the management command
[09:39] <dholbach> is there a log of the main thread?
[09:43] <davidcalle> https://pastebin.canonical.com/154786/
[09:44] <dholbach> gracias
[09:44] <dholbach> did you find the log?
[09:44] <davidcalle> https://pastebin.canonical.com/154788/
[09:45] <davidcalle> dholbach: lol -> https://pastebin.canonical.com/154789/
[09:46] <davidcalle> dholbach: I need to run, I'll add the dep as soon as I'm back. python-yaml? python3-yaml?
[09:47] <davidcalle> (not doing it now as it needs a redeploy)
[09:47] <dholbach> I don't even know where that's coming from
[09:47] <davidcalle> dholbach: the generate command
[09:47] <dholbach> ah ok
[09:48] <dholbach> python3
[09:48] <davidcalle> ok
[09:48]  * davidcalle runs, bbiab
[09:48] <dholbach> sorry about that
[09:48] <dholbach> thanks davidcalle
[09:49] <dholbach> I guess it's this:
[09:49] <dholbach> 2016-04-20 09:31:16 [17467] [CRITICAL] WORKER TIMEOUT (pid:17479)
[09:49] <dholbach> 2016-04-20 09:31:16 [17467] [CRITICAL] WORKER TIMEOUT (pid:17479)
[09:49] <dholbach> 2016-04-20 09:31:16 [30091] [INFO] Booting worker with pid: 30091
[09:49] <dholbach> can we define what a suitable worker timeout is?
[09:50] <dholbach> ok... maybe the issue here is that I'm using the admin interface
[09:50] <dholbach> and that it's going to work if we wait for the cronjob
[09:50] <dholbach> in that case we should increase the frequency
[09:50] <dholbach> and maybe go for hourly?
[09:50] <dholbach> or something
[09:50] <dholbach> let's discuss later on
[09:51] <dholbach> I'm going to meet somebody for lunch later on, so will take a longer break, but will let you know once I'm back
[12:43] <davidcalle> dholbach: back o/
[12:45] <davidcalle> dholbach: about to redeploy for the new dep... any objection?
[12:46]  * davidcalle presses button
[13:44] <dholbach> davidcalle, no, please go ahead
[13:49] <davidcalle> :)
[13:50] <dholbach> davidcalle, what do you think about what I said earlier?
[13:52] <davidcalle> dholbach: how would that solve the timeout issue, more trials per day => more chances for it to pass?
[13:52] <dholbach> no
[13:52] <dholbach> it's a different process
[13:53] <dholbach> I believe that "[CRITICAL] WORKER TIMEOUT" is a wsgi or whatever else requirement
[13:53] <dholbach> if a slightly related, but also unrelated cronjob times out that's something separate
[13:53] <dholbach> or maybe I'm wrong, I don't know
[13:54] <dholbach> mhall119's api importer takes MUCH longer to complete
[13:54] <dholbach> and it's also a cronjob
[13:54] <dholbach> not sure if I explained myself well
[13:54] <davidcalle> dholbach: I see your point about workers, but not whywe should change the cronjob (I'm not against it, though)
[13:55] <dholbach> davidcalle, using the worker through the admin interface makes no sense right no
[13:55] <dholbach> it times out and gets killed
[13:55] <davidcalle> yep
[13:55] <dholbach> this provides no easy way to test
[13:55] <dholbach> and having to wait 1 day to see results is not really practical
[13:55] <davidcalle> Oh right, for testing pruposes, gotcha
[13:55] <davidcalle> purposes*
[13:55] <dholbach> and to make sure that stuff works
[13:56] <dholbach> in normal conditions when everything works, daily is good enough for me :)
[13:57] <davidcalle> dholbach: if you make a branch running it every 30min, I can plug that in :) (currently fighting with the db dump :/, asking IS for help)
[13:57] <dholbach> shit :-/
[13:57] <dholbach> I can't believe there's no easy way to do this
[13:57] <dholbach> how are the other teams dealing with staging/prod data?
[13:59] <davidcalle> dholbach: maybe you would know? What I'm doing is dropping the db, creating it, and "psql devportal-app < dbfile", it loads fine until http://paste.ubuntu.com/15948848/
[13:59] <dholbach> maybe we can ask the U1 guys?
[13:59] <dholbach> they must have experience with this
[14:00] <davidcalle> dholbach: lol, deployment done, it worked
[14:01] <davidcalle> dholbach: somehow, after 45min of fighting, the new db is in
[14:01] <dholbach> wow
[14:01]  * davidcalle stops touching with postgre
[14:01] <dholbach> https://code.launchpad.net/~dholbach/developer-ubuntu-com/hourly-md-imports/+merge/292384
[14:01] <davidcalle> touch postgre*
[14:02] <dholbach> there must be a way to exclude certain tables from a pg dump
[14:03] <dholbach> I remember I wrote a very hacky script to exclude certain bits from a pg dump ages ago, but I could imagine that somebody did a clean job of this somewhere
[14:03] <davidcalle> dholbach: yeah, there must be one :) Sooo... Testing a few things related to the db and pushing your branch
[14:04] <davidcalle> (pushing on staging)
[14:16] <dholbach> thanks!
[14:48] <davidcalle> dholbach: mhall119 db corruption fixed on staging \o/
[14:48] <davidcalle> dpm: ^
[14:49] <dholbach> YES!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[14:49] <dholbach> __   _______ ____  _
[14:49] <dholbach> \ \ / / ____/ ___|| |
[14:49] <dholbach>  \ V /|  _| \___ \| |
[14:49] <dholbach>   | | | |___ ___) |_|
[14:49] <dholbach>   |_| |_____|____/(_)
[14:49] <dholbach>                      
[14:51] <dpm> oh wow, NICE WORK davidcalle and dholbach!
[14:51] <dholbach> ship it
[14:53] <davidcalle> dholbach: branch pushing in progress
[14:54]  * dholbach hugs davidcalle 
[15:17] <davidcalle> dholbach: pushed. You need to re-add import directives, though.
[15:19] <dholbach> awesome
[15:40] <dholbach> davidcalle, I added snappy and snapcraft docs to staging
[15:41] <dholbach> davidcalle, was python3-yaml added?
[16:01] <davidcalle> dholbach: yep
[16:01] <dholbach> davidcalle, ok... I'll take a look at it tomorrow
[16:02] <dholbach> let's see if the main imports work later on
[16:02] <dholbach> I need to run after this call
[16:27] <davidcalle> dpm: I need further confirmation from daniel, but it looks like the importer is working
[16:28] <dpm> nice!
[16:28] <davidcalle> Have a nice eod dpm o/
[16:28] <dpm> you too!