=== dendrobates is now known as dendro-afk === zyga is now known as zyga-afk === Hussain is now known as Guest32903 === zyga-afk is now known as zyga === dendro-afk is now known as dendrobates [14:29] smoser, ping.. we're seeing some issues booting instances on ec2, with apt-update failing.. was curious if you knew anything regarding http://paste.ubuntu.com/907227/ cloud-init output.. [14:30] looks like a hash mismatch on the update [14:30] utlemming, ^ [14:30] i thought the whole pipeline thing was default enabled [14:30] someone needs to open an RT on it, hazmat [14:30] we have to have IS kick that server [14:30] hazmat: run "rm -rf /var/lib/apt/lists/*" [14:30] hazmat: and then try again [14:30] * smoser is very looking forward to the hope of this all magically being fixed. [14:31] utlemming, on the instance.. we can't per se.. we rely on cloud-init to install ssh keys etc.. but its failing pretty early. [14:31] cloud-init should still install keys, no ? [14:32] * hazmat checks [14:32] smoser, it does [14:33] but manual intervention is a hoser for juju [14:33] smoser, is there an RT already? i'd be happy to file it, i'm just fuzzy on the details of whats needed [14:34] utlemming, smoser is this issue in relation to the s3 apt repos, or is it just one of the ec2 pkg repos is borked [14:34] * hazmat tries a different a region [14:34] hazmat: its the ec2 pkg repos [14:35] hazmat: so what I think is happening, is that the release file is not being updated. When "curl -I" on the Release file, the information seems stale for us-east-1. If you delete the local release file, then it should work. [14:36] hazmat: if you can verify that, then I think we'll have enough information for a RT ticket [14:36] hazmat: and I'll happily file it [14:36] utlemming, we dont really have to verify it [14:37] utlemming, i just killed the instance, but i should be able to reproduce and verify in 5m [14:38] smoser: true, but verification which proves my theory makes it easier for IS to fix it [14:44] utlemming, check-archive (i added to make it check Sources.gz and Sources.bz2) runs successfully [14:44] ./check-archive http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise [14:47] utlemming, confirmed fwiw [14:48] hazmat: okay, thank you kindly. [14:49] hazmat, what did you confirm ? [14:49] just that removing those files and trying again fixed it ? [14:51] smoser, yup [14:53] smoser: in looking at the headers between the us-west-2 and us-east-1 mirrors, the HTTP headers indicate that the us-east-1 records is stale (at least when I looked yesterday). So what I think is happening, based on what I saw yesterday is that apt is using the cached release file because it appears that a newer one doesn't exist. [14:53] smoseer: I'm rechecking now [14:55] utlemming, you're correct [14:55] smoser: see http://paste.ubuntu.com/907322/ [14:55] none of the mirrors in us-east-1 has updated since the 27th [14:57] utlemming, http://paste.ubuntu.com/907324/ [14:59] RT filed [15:06] utlemming, fwiw, this particular issue woudl have been (i think) fixed if we had the source lists in /var/lib/apt/lists === dendrobates is now known as dendro-afk === dendro-afk is now known as dendrobates === dendrobates is now known as dendro-afk === dendro-afk is now known as dendrobates === dendrobates is now known as dendro-afk