Skip to content

Feature: add option --disable-apt-cacher (following #249) #253

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AbcSxyZ wants to merge 9 commits into from
Closed

Feature: add option --disable-apt-cacher (following #249) #253

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
f9aa177
Docker apt-cacher
micaelmalta Jul 10, 2021
0b10618
make-base-vm: switch apt cacher option, using cacher enabled by default
AbcSxyZ Sep 16, 2021
f9a993e
make-base-vm: allow disabling apt cacher for lxc & kvm
AbcSxyZ Sep 16, 2021
f58f681
apt-cacher: reset default mirror and avoid use of host network with d…
AbcSxyZ Sep 16, 2021
68dd756
apt-cacher: add readme doc for --disable-apt-cacher
AbcSxyZ Sep 16, 2021
7568db8
apt-cacher: avoid use of host network with docker
AbcSxyZ Sep 16, 2021
3793b65
apt-cache: avoid use mirror with lxc
AbcSxyZ Sep 18, 2021
525e33a
apt-cacher: fix configuration error of the Dockerfile
AbcSxyZ Sep 19, 2021
d8d370d
apt-cacher: avoid configuration of bootstrap-fixup.in in make-clean-vm
AbcSxyZ Sep 20, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,13 @@ Also, I had to modify the default /etc/sudoers file to uncomment the `secure_pat

### Ubuntu:

By default and for regular build, `apt-cacher` server is used to cache dependencies. Install following packages to enable it, otherwise use `--disable-apt-cacher` :

sudo apt-get install apache2 apt-cacher-ng

This pulls in all pre-requisites for KVM building on Ubuntu:

sudo apt-get install git apache2 apt-cacher-ng python-vm-builder ruby qemu-utils
sudo apt-get install git python-vm-builder ruby qemu-utils

If you'd like to use LXC mode instead, install it as follows:

Expand All @@ -67,7 +71,7 @@ On Debian Wheezy you run the same command, but you must first add backports to y

sudo port install ruby coreutils
export PATH=$PATH:/opt/local/libexec/gnubin # Needed for sha256sum

### OSX with Homebrew:

brew install ruby coreutils
Expand Down
19 changes: 17 additions & 2 deletions bin/make-base-vm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ LXC=0
VBOX=0
DOCKER=0
DOCKER_IMAGE_HASH=""
APT_CACHER=1

usage() {
echo "Usage: ${0##*/} [OPTION]..."
Expand All @@ -24,6 +25,7 @@ usage() {
--vbox use VirtualBox instead of kvm
--docker use docker instead of kvm
--docker-image-hash D digest of the docker image to build from
--disable-apt-cacher disable APT Cacher

The MIRROR_HOST environment variable can be used to change the
apt-cacher host. It should be something that both the host and the
Expand Down Expand Up @@ -88,6 +90,10 @@ if [ $# != 0 ] ; then
DOCKER=1
shift 1
;;
--disable-apt-cacher)
APT_CACHER=0
shift 1
;;
--docker-image-digest)
DOCKER_IMAGE_HASH="$2"
shift 2
Expand Down Expand Up @@ -193,12 +199,19 @@ if [ $DOCKER = "1" ]; then
base_image="$DISTRO:$SUITE"
fi

apt_cacher=""
Copy link
Contributor Author

@AbcSxyZ AbcSxyZ Sep 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variable declaration useless

if [ "$APT_CACHER" = 1 ]; then
apt_cacher="RUN echo 'Acquire::http { Proxy \"$MIRROR_BASE\"; };' > /etc/apt/apt.conf.d/50cacher"
fi

# Generate the dockerfile
cat << EOF > $OUT.Dockerfile
FROM $base_image

ENV DEBIAN_FRONTEND=noninteractive
RUN echo 'Acquire::http { Proxy "$MIRROR_BASE"; };' > /etc/apt/apt.conf.d/50cacher
# DELETE ESM Files: W: Failed to fetch https://esm.ubuntu.com/ubuntu/dists/trusty-infra-security/main/binary-amd64/Packages Received HTTP code 403 from proxy after CONNECT
RUN [ -f /etc/apt/sources.list.d/*esm*.list ] && rm /etc/apt/sources.list.d/*esm*.list
$apt_cacher
RUN apt-get update && apt-get --no-install-recommends -y install $addpkg

RUN useradd -ms /bin/bash -U $DISTRO
Expand Down Expand Up @@ -255,6 +268,7 @@ if [ $LXC = "1" ]; then
echo "sudo will preserve (some) env flags"
preserve_env=yes # if you would want to set false then unset this variable
fi
[ $APT_CACHER -eq 0 ] && MIRROR=""
env -i LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 DEBOOTSTRAP_DIR="$DEBOOTSTRAP_DIR" sudo ${preserve_env+--preserve-env} debootstrap --arch=$ARCH --include=$addpkg --exclude=$removepkg --components=$components $SUITE $OUT-bootstrap $MIRROR
# Fix lxc issue
if [ -f $OUT-bootstrap/usr/lib/lxc/lxc-init ]
Expand Down Expand Up @@ -293,7 +307,8 @@ else
libexec/config-bootstrap-fixup

rm -rf $OUT
env -i LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 sudo vmbuilder kvm $DISTRO --rootsize $DISKSIZE --arch=$ARCH --suite=$SUITE --addpkg=$addpkg --removepkg=$removepkg --ssh-key=var/id_rsa.pub --ssh-user-key=var/id_rsa.pub --mirror=$MIRROR --security-mirror=$SECURITY_MIRROR --dest=$OUT --flavour=$FLAVOUR --firstboot=`pwd`/target-bin/bootstrap-fixup
[ $APT_CACHER -eq 1 ] && mirror_options="--mirror=$MIRROR --security-mirror=$SECURITY_MIRROR"
env -i LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 sudo vmbuilder kvm $DISTRO --rootsize $DISKSIZE --arch=$ARCH --suite=$SUITE --addpkg=$addpkg --removepkg=$removepkg --ssh-key=var/id_rsa.pub --ssh-user-key=var/id_rsa.pub --dest=$OUT --flavour=$FLAVOUR --firstboot=`pwd`/target-bin/bootstrap-fixup $(echo $mirror_options)
mv $OUT/*.qcow2 $OUT.qcow2
rm -rf $OUT
# bootstrap-fixup is done on first boot
Expand Down
6 changes: 5 additions & 1 deletion libexec/config-bootstrap-fixup
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,8 @@ if [ -z "$MIRROR_HOST" ] || [ "$MIRROR_HOST" == "127.0.0.1" ]; then
MIRROR_HOST=$GITIAN_HOST_IP
fi

sed "s;HOSTIP;$MIRROR_HOST;g" < target-bin/bootstrap-fixup.in > target-bin/bootstrap-fixup
if [ $APT_CACHER = "1" ]; then
Copy link
Owner

@devrandom devrandom Nov 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

two issues:

  • missing quotes around $APT_CACHER
  • APT_CACHER is not actually exported as an environment variable

sed "s;HOSTIP;$MIRROR_HOST;g" < target-bin/bootstrap-fixup.in > target-bin/bootstrap-fixup
else
sed "s;HOSTIP:3142/;;g" < target-bin/bootstrap-fixup.in > target-bin/bootstrap-fixup
fi
Comment on lines +15 to +19
Copy link
Contributor Author

@AbcSxyZ AbcSxyZ Sep 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually not working.

config-boostrap-fixup is called with bin/make-base-vm, which have APT_CACHER variable through --disable-apt-cacher.
It is called also with bin/gbuild, who use it with make-clean-vm. At this moment, they do not have APT_CACHER variable anymore.

Creating the following error :

bash: [: =: unary operator expected

APT_CACHER will be replaced by an unquoted empty value.

Copy link
Contributor Author

@AbcSxyZ AbcSxyZ Sep 20, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a possible solution, would it be possible to configure bootstrap-fixup.in only when using make-base-vm ? And then reuse generated bootstrap-fixup file instead of recreating one in make-clean-vm.

The feature was implemented by commit 5785dfc, I suppose we can do it as I suggest in the case GITIAN_HOST_IP doesn't change between make-base-vm & gbuild.
Actually, without this situation, it looks like it's creating twice the same file.

It can make sure configuration is more done through make-base-vm.
Maybe deleting the following line can do the job (?) :

gitian-builder/libexec/make-clean-vm

Line 65 in 4609325

libexec/config-bootstrap-fixup

Copy link
Contributor Author

@AbcSxyZ AbcSxyZ Sep 20, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if it's the right solution, but if it is I just added the commit d8d370d to implement this fix.

1 change: 0 additions & 1 deletion libexec/make-clean-vm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,6 @@ case $VMSW in
;;
LXC)
cp -a --sparse=always $BASE $OUT
libexec/config-bootstrap-fixup
on-target -u root bash < target-bin/bootstrap-fixup
;;
VBOX)
Expand Down