| kb9vqf | wgrant: ping | 03:43 |
|---|---|---|
| kb9vqf | I have the production system upgraded to the latest source and have been squashing remaining bugs but the build farm itself is giving me quite a bit of grief | 03:44 |
| kb9vqf | I have a test package that's built but stuck "Uploading build" | 03:44 |
| kb9vqf | If I understand correctly this means the builder already uploaded the built files but the script responsible for publishng them is not running properly? | 03:45 |
| kb9vqf | I thought process-accepted.py was responsible for publishing the uploaded build files, but now I am not so sure | 03:45 |
| kb9vqf | Help? :-) | 03:45 |
| * kb9vqf is also trying to reenable the build farm on non-virtualized builders...seems quite a lot changed over 4 years | 03:46 | |
| kb9vqf | Hmmm...finally came across a page detailing some of the upload process (https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally). I think poppy isn't even receiving the packages; let me dig a bt further | 03:50 |
| wgrant | kb9vqf: Ah, I forget when some of this changed. | 03:51 |
| wgrant | kb9vqf: build uploads don't actually use poppy. | 03:51 |
| kb9vqf | Ah, ok. | 03:51 |
| kb9vqf | Where do they go? | 03:51 |
| kb9vqf | I don't even see any log files | 03:51 |
| wgrant | They should end up automatically somewhere like /var/tmp/buildmaster/incoming | 03:51 |
| wgrant | That path will be logged in the buildd-manager log. | 03:52 |
| * kb9vqf goes to look | 03:52 | |
| wgrant | alphecca.canonical.com-lp_buildd:* * * * * /srv/launchpad.net/codelines/current/scripts/process-upload.py -C buildd --builds /srv/launchpad.net/builddmaster/ -q --log-file=DEBUG:/srv/launchpad.net/production-logs/process-build-uploads.log | 03:52 |
| wgrant | That's how we process them on production. | 03:52 |
| kb9vqf | at least the builder is sort of working: "Grabbing file:"... | 03:52 |
| wgrant | That changed a couple of times in 2010. | 03:52 |
| wgrant | If it got to "Uploading build", the builder worked fine and it's in the upload queue. | 03:52 |
| kb9vqf | ok, let me check a couple things real quick | 03:53 |
| wgrant | You just need to find where the upload queue is, and ensure process-upload -C buildd --builds is running over it. | 03:53 |
| kb9vqf | yeop, my process-upload is running againast poppy | 03:53 |
| kb9vqf | *against | 03:53 |
| wgrant | You need both. | 03:53 |
| kb9vqf | gotcha | 03:53 |
| wgrant | One with -C insecure for the untrusted source uploads from poppy, and one with -C buildd --builds for the trustworthy binary uploads from buildd-manager. | 03:53 |
| wgrant | (though -C insecure might be the default; I forget) | 03:54 |
| * kb9vqf confirms the second was missing | 03:54 | |
| kb9vqf | the 2010 timeframe sounds right | 03:54 |
| wgrant | buildd-manager used to do that inline, but it doesn't scale very well when you're processing jobs from 150 buildds. | 03:54 |
| wgrant | So it just writes them to a normal upload queue and waits for a cronjob nowadays. | 03:54 |
| kb9vqf | that should help--I upgraded because the build farm was not scaling well | 03:55 |
| kb9vqf | so glad to see that improvements were made on that end :-) | 03:55 |
| kb9vqf | Any ideas though on what to do about this error: Scanning eostre failed with: Allegedly clean slave not idle ('BuilderStatus.WAITING' instead) | 03:55 |
| kb9vqf | I can't virtualize the farm, and am contemplating hacking the buildd-manager to force it to work on my setub, but want to make sure there isn't flag I can set instead | 03:56 |
| kb9vqf | wow my keyboard is behaving badly tonight | 03:56 |
| kb9vqf | sorry :-P | 03:56 |
| wgrant | Hmmm. | 03:57 |
| kb9vqf | basically it builds one package and uploads it, cleans the slave and then dies | 03:57 |
| wgrant | Yeah | 03:57 |
| wgrant | So all your builders are marked virtualized in Launchpad, but aren't actually reset between builds because they're real hardware? | 03:58 |
| kb9vqf | Right | 03:58 |
| wgrant | We used to do that for ARM too, but not any more so I added some safety checks. | 03:58 |
| kb9vqf | Yep arm here is the problem | 03:58 |
| wgrant | You have two options. | 03:58 |
| wgrant | a) Set all the builders and the relevant PPAs to non-virtual, and hack the database to ensure the existing builds are non-virtual (trivial). | 03:59 |
| wgrant | Or b) Adjust lib/lp/buildmaster/interactor.py's cleanSlave method to always use the non-virtual path. | 04:00 |
| * kb9vqf thinks b) would be best as a) causes user-visible changes | 04:00 | |
| wgrant | The virtualized path is optimised to not bother cleaning after a build, since obvious a VM reset is going to result in a clean VM. | 04:00 |
| kb9vqf | you guys still using Xen? | 04:00 |
| wgrant | We switched to an KVM+OpenStack solution in August. | 04:01 |
| kb9vqf | ok, that sounds better | 04:01 |
| kb9vqf | kvm works on arm IIRC so that might even be an option now | 04:01 |
| wgrant | https://insights.ubuntu.com/2014/10/30/scalingstack-2x-performance-in-launchpads-build-farm-with-openstack/ outlines the old and new solutions. | 04:01 |
| kb9vqf | great! I'll give that a read | 04:01 |
| wgrant | But I would recommend a) for your case. It shouldn't cause any user-visible changes unless you're doing very weird things. | 04:01 |
| kb9vqf | first thing that happens is the build farm status page shows all the non-virtualized builders | 04:02 |
| wgrant | Well, they just use a different icon and move to a slightly different place on the page. | 04:02 |
| wgrant | Is that a problem? | 04:02 |
| kb9vqf | switching the builders to non-virtual puts them into the "officia" build queue and leaves "0 builders" for PPAs | 04:02 |
| kb9vqf | not a big problem, but ugly | 04:03 |
| wgrant | Yeah, we've needed to change those descriptions for a while, but to avoid confusing users in the normal case we've left them that way. | 04:03 |
| kb9vqf | actually if you're on KVM now I might look into the virtualization | 04:03 |
| wgrant | You could easily hack the template to only show one set of stats, though. | 04:03 |
| wgrant | Our ARM stuff is still unvirtualised. | 04:03 |
| kb9vqf | It was a no-go on Xen due to arm but now... | 04:03 |
| wgrant | ARM PPAs use qemu-user :/ | 04:03 |
| kb9vqf | oh, ok | 04:03 |
| wgrant | But if you have supportable hardware, it works. | 04:03 |
| kb9vqf | real arm hardware here | 04:04 |
| wgrant | Sadly virt-capable server-class ARM hardware is just now coming available. | 04:04 |
| wgrant | You need at least Cortex-A15. | 04:04 |
| wgrant | An A8 or A9 will not do. | 04:04 |
| kb9vqf | Cortex A9 here so no-go | 04:04 |
| wgrant | Yeah :/ | 04:04 |
| kb9vqf | well looks like an A15 upgrade will be needed in the future then | 04:04 |
| wgrant | We hope to move to a fully virtualized pool within a year or so. | 04:04 |
| wgrant | Once we have enough virt-capable ARM hardware in the pool. | 04:04 |
| wgrant | Currently the non-virtualised ARM buildds are quad-A9s. | 04:05 |
| kb9vqf | I was hoping I could virtualize as I get tired of having the build pool go offline under heavy load | 04:05 |
| kb9vqf | but oh wll | 04:05 |
| kb9vqf | *well | 04:05 |
| kb9vqf | Yeah, quad A9s here too | 04:05 |
| * kb9vqf wonders if we're using the same chip :) | 04:05 | |
| kb9vqf | wgrant: upload accepted, thanks for the help! :-) | 04:09 |
| * kb9vqf wonders if he should update that Wiki page | 04:09 | |
| wgrant | kb9vqf: Excellent. | 04:10 |
| * kb9vqf reads the Wiki again and sees the info clearly there--must have been stuck on the old system too long :-P | 04:10 | |
| wgrant | kb9vqf: Oh, also, manage-chroot.py is no longer a script in the LP tree. It's an API client in lp:ubuntu-archive-tools. | 04:10 |
| kb9vqf | that would have tripped me up; thanks for the info | 04:11 |
| wgrant | And if you want to create non-Ubuntu PPAs, you need to use the API. The +active-ppa form still forces Ubuntu today. | 04:11 |
| wgrant | s/active-ppa/activate-ppa/ | 04:11 |
| kb9vqf | wgrant: for now we'll just continue with our hack; no need to change whats working until the other solution is better | 04:11 |
| wgrant | Yep. | 04:11 |
| kb9vqf | wgrant: calling convention for the new manage-chroot API client is documented somewhere? | 04:12 |
| wgrant | kb9vqf: Should be pretty much the same as before, IIRC. It should also be on that page, and in --help | 04:12 |
| kb9vqf | ok, just thought I'd verify while you were on the line so to speak | 04:12 |
| wgrant | You'll need to give it something like '-l https://api.quickbuild.pearsoncomputing.net/' or so as well, though. | 04:13 |
| kb9vqf | sure, makes sense | 04:13 |
| wgrant | The manage-builders script in the same tree is probably also of interest for bulk operations on the build farm. | 04:13 |
| kb9vqf | that would be very nice--I was looking into writing something like that | 04:13 |
| kb9vqf | wgrant: one more little question...I noticed the Google Maps stuff was ripped out. We were using that and I kind of miss it; any chance of it coming back? I filed a bug report on it yesterday. | 04:15 |
| wgrant | kb9vqf: It was removed a few years ago because Google Maps was no longer unusable for various reasons. It would probably be possible to reimplement it on top of OpenStreetMap, but it's not something we're interested in today. | 04:18 |
| kb9vqf | ok | 04:18 |
| kb9vqf | wgrant: is there a document available that details the codehosting setup/how to get bzr imports working? I am looking at activating it but immediately see references to internal Canonical machines running unknown software | 04:20 |
| wgrant | kb9vqf: Codehosting is a fair bit of work to set up and maintain. I'd avoid it unless you have no other choice, particularly given that things will be radically changing there over the next few months. | 04:22 |
| kb9vqf | OK, we can hold off. Changing for the better I hope? | 04:22 |
| wgrant | That's the plan. | 04:22 |
| kb9vqf | :-) | 04:22 |
| kb9vqf | What I'd really like to see is Launchpad's translation system interface with a GIT module, but I don't know if that's possible | 04:23 |
| kb9vqf | s/module/repository/ | 04:23 |
| kb9vqf | Well I converted the database over to allow non-virtualized builds...test builder is active, let's see how it goes :-) | 04:30 |
| * wgrant watches. | 04:31 | |
| kb9vqf | wgrant: Does the new PPA creation code force require_virtualized to True or does it use the database default? | 04:32 |
| wgrant | kb9vqf: That hasn't changed. They always default to virtualized. | 04:32 |
| wgrant | You can tweak it per-PPA on +admin, and should probably leave the default alone for security reasons, unless you really need to create lots of PPAs regularly. | 04:33 |
| wgrant | Given that anyone who can log in can create a PPA. | 04:33 |
| kb9vqf | wgrant: Right, I wasn't clear. I mean since I changed the database to default to non-virtualized do I need to also change the code? I have users that do create PPAs and I don't want to go into the database constantly to allow builds | 04:33 |
| wgrant | kb9vqf: Yes. If you really want to change that default, despite the security implications, you'll probably want to change Person.createPPA to call ArchiveSet.new with require_virtualized=False | 04:36 |
| kb9vqf | Thanks, that's exactly what I needed to know | 04:36 |
| kb9vqf | I've been running it this way for all these years with no problems (I dealt with the security holes in the infrastrucutre) | 04:37 |
| kb9vqf | *infrastructure | 04:37 |
| * kb9vqf notes with some satisfaction that the previously uploaded test build has now been fully published to the archive. Still waiting on non-virtualized build tests. | 04:38 | |
| wgrant | Excellent. | 04:42 |
| kb9vqf | So aside from codehosting any other major changes coming down that I should be aware of? I won't be using the bugtracker at any point as we have a well-entrenched Bugzilla but some of the other features are interesting | 04:43 |
| wgrant | Nothing scheduled right now that should make your life hell. | 04:44 |
| kb9vqf | Sounds good | 04:44 |
| wgrant | Though there will probably be some Soyuz changes next year some time that might have ramifications, they'll be nothing that's terribly difficult to migrate to. | 04:44 |
| kb9vqf | Ah, this wasn't hell, it was just a .... challenging .... upgrade with .... lots of downtime .... hmmm, on second thought let's not repeat it :-) | 04:45 |
| wgrant | Though I am doing a BinaryPackageBuild schema change now that has a Python migration, but I'll include an SQL patch to do the same thing :) | 04:45 |
| kb9vqf | thanks for thinking of the only other Launchpad instance; I appreciate it :-) | 05:20 |
| kb9vqf | so far the non-virtualized test builder is working perfectly, even with build failures, so I'm going to cross my fingers and activate additional builders | 05:23 |
| wgrant | Lovely. | 05:24 |
| kb9vqf | base system looks to be running well (much faster than the old Launchpad instance actually); I'm going to sign off for tonight (wee hours of the morning here). Thanks again! | 07:14 |
| === timrc-afk is now known as timrc | ||
| === timrc is now known as timrc-afk | ||
| === timrc-afk is now known as timrc | ||
| kb9vqf | wgrant: Just ran into a problem with a builder that was functional until the builds were accidentally interrupted | 23:05 |
| kb9vqf | log now states "Build log: /home/buildd/.lp-sbuildrc is corrupt; builder needs repair work" | 23:05 |
| kb9vqf | have you run into this before? | 23:05 |
| kb9vqf | the file mentioned doesn't exist on the system | 23:06 |
| kb9vqf | reinstalling the package fixed it, nevermind :) | 23:07 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!