• Step 5 (option B): Configure buildroot

    andriy.malyshenko • 08/13/2021 at 09:34 • 0 comments

    In the last step we're use Armbian distribution to take snapshot of the root filesystem, as well as boot artifacts.

    Alternatively to that we can generate root filesystem as well as boot artifacts using [buildroot](https://github.com/buildroot/buildroot)

    In short this is a set of tools and scripts that allows you to generate and build small yet extremely powerfull (and very much configurable) linux distribution. It will build u-boot (we don't need that), kernel and root filesystem specifically for our Orange Pi.

    Execute following step on the dev machine

    $ cd ~/pi-boot/
$ git clone https://github.com/anabolyc/buildroot-orange-pi-zero && cd buildroot-orange-pi-zero
$ git submodule update --init
$ cd buildroot
$ make orangepi_zero_defconfig
# optional, but you might want to configure root password, host name or something else to your taste
$ make menuconfig
$ make

    This will take a while, and in the end it will output 

    [...]

INFO: vfat(boot.vfat): cmd: "dd if=/dev/zero of="/media/dronische/storage/buildroot-orange-pi-zero/buildroot/output/images/boot.vfat" seek=10485760 count=0 bs=1 2>/dev/null" (stderr):
INFO: vfat(boot.vfat): cmd: "mkdosfs   '/media/dronische/storage/buildroot-orange-pi-zero/buildroot/output/images/boot.vfat'" (stderr):
INFO: vfat(boot.vfat): adding file 'zImage' as 'zImage' ...
INFO: vfat(boot.vfat): cmd: "MTOOLS_SKIP_CHECK=1 mcopy -bsp -i '/media/dronische/storage/buildroot-orange-pi-zero/buildroot/output/images/boot.vfat' '/media/dronische/storage/buildroot-orange-pi-zero/buildroot/output/images/zImage' '::'" (stderr):
INFO: vfat(boot.vfat): adding file 'sun8i-h2-plus-orangepi-zero.dtb' as 'sun8i-h2-plus-orangepi-zero.dtb' ...
INFO: vfat(boot.vfat): cmd: "MTOOLS_SKIP_CHECK=1 mcopy -bsp -i '/media/dronische/storage/buildroot-orange-pi-zero/buildroot/output/images/boot.vfat' '/media/dronische/storage/buildroot-orange-pi-zero/buildroot/output/images/sun8i-h2-plus-orangepi-zero.dtb' '::'" (stderr):
INFO: vfat(boot.vfat): adding file 'boot.scr' as 'boot.scr' ...
INFO: vfat(boot.vfat): cmd: "MTOOLS_SKIP_CHECK=1 mcopy -bsp -i '/media/dronische/storage/buildroot-orange-pi-zero/buildroot/output/images/boot.vfat' '/media/dronische/storage/buildroot-orange-pi-zero/buildroot/output/images/boot.scr' '::'" (stderr):
INFO: hdimage(sdcard.img): adding partition 'u-boot' from 'u-boot-sunxi-with-spl.bin' ...
INFO: hdimage(sdcard.img): adding partition 'boot' (in MBR) from 'boot.vfat' ...
INFO: hdimage(sdcard.img): adding partition 'rootfs' (in MBR) from 'rootfs.ext4' ...
INFO: hdimage(sdcard.img): writing MBR

    Now we have everything we need locally, just need to copy that to both NFS and TFTP servers

    $ mkdir -p ~/pi-boot/mnt/{sdc,nfs} && cd ~/pi-boot
$ sudo mount 192.168.1.3:/srv/exports mnt/nfs
$ sudo mkdir mnt/nfs/01-02-42-94-b4-99-28-buildroot
$ ls -la mnt/nfs
total 16
drwxr-xr-x  4 root      root      4096 sie 13 10:25 .
drwxrwxr-x  4 dronische dronische 4096 sie 13 08:52 ..
drwxr-xr-x 18 root      root      4096 sie 13 09:54 01-02-42-94-b4-99-28
drwxr-xr-x 16 root      root      4096 sie 12 21:10 01-02-42-94-b4-99-28-buildroot

$ sudo rsync -va ~/pi-boot/buildroot-orange-pi-zero/buildroot/output/staging/ mnt/nfs/01-02-42-94-b4-99-28-buildroot

    Next lets copy kernel and device tree to TFTP share

    $ scp ~/pi-boot/buildroot-orange-pi-zero/buildroot/output/images/{zImage,sun8i-h2-plus-orangepi-zero.dtb} 192.168.1.2:/srv/tftp

    And edit boot config (from TFTP server now)

    $ cd /srv/tftp
$ sudo mkdir dtb-buildroot && mv sun8i-h2-plus-orangepi-zero.dtb ./dtb-buildroot
$ sudo nano pxelinux.cfg/01-02-42-94-b4-99-28

     Content of the file

    LABEL linux
KERNEL vmlinuz-5.10.43-sunxi
FDTDIR dtb-5.10.43-sunxi
APPEND root=/dev/nfs initrd=uInitrd-5.10.43-sunxi nfsroot=192.168.1.3:/srv/exports/01-02-42-94-b4-99-28 ip=dhcp rw

LABEL buildroot
KERNEL zImage
FDTDIR dtb-buildroot
APPEND root=/dev/nfs nfsroot=192.168.1.3:/srv/exports/01-02-42-94-b4-99-28-buildroot,vers=4 ip=dhcp rw

DEFAULT buildroot

    Now let's reboot our PI 

    U-Boot SPL 2020.10-armbian...
    Read more »

  • Step 5: Configure NFS server and finish TFTP config

    andriy.malyshenko • 08/13/2021 at 08:14 • 0 comments

    Now we need to prepare Pi specific files. There are number of ways doing this. Simplest one is to take files from SD card that we have prepared earlier. But first let's prepare last server in the setup - NFS server. This will provide root filesystem fro our Pi over network and preferably should have fast storage and wide connection to PI. But of course that is only optional.

    In my case it is ubuntu machine under 192.168.1.3 IP, but it can be combined with TFTP and/or DHCP server as well.

    $ sudo apt install nfs-kernel-server -y
$ sudo mkdir -p /srv/exports/01-02-42-94-b4-99-28
$ sudo nano /etc/exports

    Contents of  exports config:

    /srv/exports 192.168.1.0/24(rw,no_root_squash,async,insecure,no_subtree_check,crossmnt)

    You can play around with settings, above those working best for me. Key issue you might address is that all machines in the network are allowed to connect to that NFS export without authorisation. This is okay for me, but you might want to have different setup, perhaps not as much trusted.

    Now lets update service

    $ sudo exportfs -rv

    Now we're back at dev machine, where we can take snapshot from SD card

    $ mkdir -p ~/pi-boot/mnt/{sdc,nfs} && cd ~/pi-boot
$ sudo mount /dev/sdc1 mnt/sdc
$ sudo mount 192.168.1.3:/srv/exports mnt/nfs
$ ls -la mnt/nfs/
total 12
drwxr-xr-x 3 root      root      4096 sie 13 08:49 .
drwxrwxr-x 4 dronische dronische 4096 sie 13 08:52 ..
drwxr-xr-x 2 root      root      4096 sie 13 08:49 01-02-42-94-b4-99-28

$ sudo rsync -va mnt/sdc/ mnt/nfs/01-02-42-94-b4-99-28

sent 987,891,328 bytes  received 708,418 bytes  6,009,724.90 bytes/sec
total size is 985,196,807  speedup is 1.00

    Few small fixes that we need to apply on the root filesystem

    # we no longer need this, it is stored on TFTP now
$ sudo rm -rf mnt/nfs/01-02-42-94-b4-99-28/boot

# edit fstab
$ sudo nano mnt/nfs/01-02-42-94-b4-99-28/etc/fstab

    Need to comment first line, since we no longer have SD-card

    # UUID=3dbf7aef-312c-456c-a992-e6bde7857d3f / ext4 defaults,noatime,commit=600,errors=remount-ro 0 1
tmpfs /tmp tmpfs defaults,nosuid 0 0

      Next we need to prepare boot artifacts on TFTP server

    $ cd ~/pi-boot/mnt/sdc/boot
$ scp -r uInitrd-5.10.43-sunxi vmlinuz-5.10.43-sunxi dtb-5.10.43-sunxi 192.168.1.2:/srv/tftp

    Note that names above are those specified in boot config file  created at previous step.


    Last thing is to restore permissions on the TFTP share

    $ sudo chown root:root /srv/tftp -R

    New let's try to boot Pi

    Autoboot in 1 seconds
MMC: no card present

Device 0: unknown device
ethernet@1c30000 Waiting for PHY auto negotiation to complete. done
BOOTP broadcast 1
DHCP client bound to address 192.168.1.49 (9 ms)
Using ethernet@1c30000 device
TFTP from server 192.168.1.2; our IP address is 192.168.1.49
Filename '/pxelinux.0'.
Load address: 0x42000000
Loading: *
TFTP error: 'File not found' (1)
Not retrying...
missing environment variable: pxeuuid
Retrieving file: /pxelinux.cfg/01-02-42-94-b4-99-28
Using ethernet@1c30000 device
TFTP from server 192.168.1.2; our IP address is 192.168.1.49
Filename '/pxelinux.cfg/01-02-42-94-b4-99-28'.
Load address: 0x43200000
Loading: #
         63.5 KiB/s
done
Bytes transferred = 195 (c3 hex)
Config file found
1:      linux
Retrieving file: /uInitrd-5.10.43-sunxi
Using ethernet@1c30000 device
TFTP from server 192.168.1.2; our IP address is 192.168.1.49
Filename '/uInitrd-5.10.43-sunxi'.
Load address: 0x43300000
Loading: #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
 #################################################################
...
    Read more »

  • Step 4: Configure TFTP server

    andriy.malyshenko • 08/13/2021 at 06:35 • 0 comments

    When Pi boots it will request certain files from TFTP server. It is very simple server with no authentication and it only server few files, among which kernel, initrd and boot config.

    First we will configure some TFTP server, then we will upload those resources.

    So next commands are executed on 192.168.1.2 machine, which is ubuntu in my case

    $ sudo apt install tftpd-hpa -y
$ sudo nano /etc/default/tftpd-hpa
$ sudo chown $UID:$UID /srv/tftp -R
$ mkdir /srv/tftp/pxelinux.cfg
$ nano /srv/tftp/pxelinux.cfg
$ nano /srv/tftp/pxelinux.cfg/01-02-42-94-b4-99-28

    Content of that file is

    LABEL linux
KERNEL vmlinuz-5.10.43-sunxi
FDTDIR dtb-5.10.43-sunxi
APPEND root=/dev/nfs initrd=uInitrd-5.10.43-sunxi nfsroot=192.168.1.3:/srv/exports/01-02-42-94-b4-99-28 ip=dhcp rw
DEFAULT linux

    As you can see above, configuration is per MAC address, so I can have multiple device configurations in the network.

    For now we don't have yet kernel, initrd and dtb files, we will fix that in a minute. Also we inform that NFS server will be under 192.168.1.3 address which not exists yet.

    In the Pi boot log we're one step closer

    [...]

Autoboot in 1 seconds
MMC: no card present
   
Device 0: unknown device
ethernet@1c30000 Waiting for PHY auto negotiation to complete. done
BOOTP broadcast 1
DHCP client bound to address 192.168.1.49 (8 ms)
Using ethernet@1c30000 device
TFTP from server 192.168.1.2; our IP address is 192.168.1.49
Filename '/pxelinux.0'.
Load address: 0x42000000
Loading: *
TFTP error: 'File not found' (1)
Not retrying...
missing environment variable: pxeuuid
Retrieving file: /pxelinux.cfg/01-02-42-94-b4-99-28
Using ethernet@1c30000 device
TFTP from server 192.168.1.2; our IP address is 192.168.1.49
Filename '/pxelinux.cfg/01-02-42-94-b4-99-28'.
Load address: 0x43200000
Loading: #
         43.9 KiB/s
done
Bytes transferred = 183 (b7 hex)
Config file found
1:      linux
Retrieving file: /initrd.img-5.10.43-sunxi

[...]

    Next we'll prepare files specific for Pi both on TFTP server and NFS server

  • Step 3: Configure DHCP server

    andriy.malyshenko • 08/12/2021 at 21:17 • 0 comments

    Now when our Pi asking to boot from network we need someone to answer it. First to respond will be DHCP server, whose role is to assign IP address (I prefer static one, but it is not strictly necessary) and instruct to get boot resources from TFTP server.

    I assume you already have some kind of DHCP server within your network, normally that would be a network router, but might be dedicated machine. Im my case it is running [OpenWrt](https://openwrt.org/) and I'm able to ssh in it.

    Following changes are executed at router filesystem.

    # nano /etc/config/dhcp

    Add below config to the bottom

    config boot 'linux'
	option filename '/pxelinux.0'
	option serveraddress '192.168.1.2'
	option servername 'netbootsrv'

config host                           
        option name 'opi-zero-z4'          
        option mac '02:42:94:b4:99:28'
        option ip '192.168.1.48'

    For boot section we define basic structure that expected to exist on TFTP server and actual IP of netboot server, which is 192.168.1.2 in my case.

    For Pi section ame is friendly name of your pi, mac is unique HW address of your Pi (I look for it in DHCP logs, but there are number of ways to find it). IP address is up to you, you just need to remember it for the future steps.


    If we boot Pi now

    [...]

Autoboot in 1 seconds
MMC: no card present

Device 0: unknown device
ethernet@1c30000 Waiting for PHY auto negotiation to complete. done
BOOTP broadcast 1
DHCP client bound to address 192.168.1.49 (6 ms)
Using ethernet@1c30000 device
TFTP from server 192.168.1.2; our IP address is 192.168.1.49
Filename '/pxelinux.0'.
Load address: 0x42000000

    Both of our settings are accepted, but TFTP server is not responding, that what we're going to fix in next step

  • Step 2: enable SPI flash and and write u-boot

    andriy.malyshenko • 08/11/2021 at 22:30 • 0 comments

     Next step is to flash u-boot (ie bootloader) to SPI flash.

    When Pi starts normally first stage bootloader (this one is hard-coded into the CPU itself) will look for boot device. Normally it will find sd-card and boot from it (it has highest priority).

    When it is not present however it will try luck with other boot sources, including SPI flash. So it will actually initialize SPI bus all on it's own and look for bootloader there (u-boot in our case). So in this step we will flash it there. Alternatively you may programm SPI flash directly on the Pi, but this is more advanced scenario and it is not covered in this tutorial.

    Next steps are run on Pi

    $ sudo apt update 
$ sudo apt upgrade
$ sudo apt-get install flashrom

     Next we will enable SPI flash to gain access to it (it is disabled by default) by adding device tree layer named spi-spidev

    $ sudo nano /boot/armbianEnv.txt

     File will look like this after edit

    verbosity=1
bootlogo=false
console=serial
disp_mode=1920x1080p60
overlay_prefix=sun8i-h3
overlays=usbhost2 usbhost3 spi-spidev
rootdev=UUID=3dbf7aef-312c-456c-a992-e6bde7857d3f
rootfstype=ext4
usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u
param_spidev_spi_bus=0

     We reboot after that to confirm presense of SPI device

    $ sudo reboot

[...]

$ ls -l /dev/spi*
crw------- 1 root root 153, 0 Aug 11 22:19 /dev/spidev0.0

    Device is present, we can start preparing image to flash.

    $ mkdir spiflash && cd spiflash

# create empty image, ouf flash chip can be larger, but for our purposes it is just enough
$ dd if=/dev/zero count=2048 bs=1K | tr '\000' '\377' > spi.img

# let's find u-boot binary in the local filesystem
$ ls -al /usr/lib/linux-u-boot-*_armhf/*.bin 
-rw-rw-r-- 1 root root 469855 Aug  8 14:21 /usr/lib/linux-u-boot-current-orangepizero_21.08.1_armhf/u-boot-sunxi-with-spl.bin

$ dd if=/usr/lib/linux-u-boot-current-orangepizero_21.08.1_armhf/u-boot-sunxi-with-spl.bin of=spi.img bs=1k conv=notrunc

    Now we have an image in spi.img file, that we just need to flash 

    $ sudo flashrom -p linux_spi:dev=/dev/spidev0.0 -w spi.img -c MX25L1605A/MX25L1606E/MX25L1608E

flashrom v1.2 on Linux 5.10.43-sunxi (armv7l)
flashrom is free software, get the source code at https://flashrom.org

Using clock_gettime for delay loops (clk_id: 1, resolution: 1ns).
Using default 2000kHz clock. Use 'spispeed' parameter to override.
Found Macronix flash chip "MX25L1605A/MX25L1606E/MX25L1608E" (2048 kB, SPI) on linux_spi.
Reading old flash chip contents... done.
Erasing and writing flash chip... Erase/write done.
Verifying flash... VERIFIED.

    Now we're ready to reboot and try out boot without SD-card present
     

    $ sudo poweroff

     Upon reboot without SD-card

    U-Boot SPL 2020.10-armbian (Aug 08 2021 - 16:21:05 +0200)
DRAM: 256 MiB
Trying to boot from sunxi SPI
   
   
U-Boot 2020.10-armbian (Aug 08 2021 - 16:21:05 +0200) Allwinner Technology
   
CPU:   Allwinner H3 (SUN8I 1680)
Model: Xunlong Orange Pi Zero
DRAM:  256 MiB
MMC:   mmc@1c0f000: 0, mmc@1c10000: 1
Loading Environment from FAT... MMC: no card present
In:    serial
Out:   serial
Err:   serial
Net:   phy interface0
eth0: ethernet@1c30000
starting USB...
Bus usb@1c1a000: USB EHCI 1.00
Bus usb@1c1a400: USB OHCI 1.0
Bus usb@1c1b000: USB EHCI 1.00
Bus usb@1c1b400: USB OHCI 1.0
Bus usb@1c1c000: USB EHCI 1.00
Bus usb@1c1c400: USB OHCI 1.0
scanning bus usb@1c1a000 for devices... 1 USB Device(s) found
scanning bus usb@1c1a400 for devices... 1 USB Device(s) found
scanning bus usb@1c1b000 for devices... 1 USB Device(s) found
scanning bus usb@1c1b400 for devices... 1 USB Device(s) found
scanning bus usb@1c1c000 for devices... 1 USB Device(s) found
scanning bus usb@1c1c400 for devices... 1 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
Autoboot in 1 seconds
MMC: no card present
   
Device 0: unknown device
ethernet@1c30000 Waiting for PHY auto negotiation to complete. done
BOOTP broadcast 1
DHCP client bound to address 192.168.1.202 (8 ms)

     What happens above is Pi able...

    Read more »

  • Step 1: prepare SD-card image, boot, perform initial configuration

    andriy.malyshenko • 08/11/2021 at 21:56 • 0 comments

    First we would need bootable SD-card. As mentioned before we will use it to (a) flash onboard SPI and (b) take snapshot of the root FS.

    1. I'll start with Armbian, later on we might want to take another distro. Go to Armbian website and get latest image, at the moment of writing it is Armbian_21.05.6_Orangepizero_focal_current_5.10.43.img.xz

    2. Insert SD card and note its name. It is sdc in my case. No need to mount or format it.

    $ lsblk
[...]
sdc                     8:32   1  14,6G  0 disk  
└─sdc1                  8:33   1  14,5G  0 part

     3. Flash the image to SD card

    $ xz --decompress Armbian_21.05.6_Orangepizero_focal_current_5.10.43.img.xz
$ sudo dd if=Armbian_21.05.6_Orangepizero_focal_current_5.10.43.img of=/dev/sdc bs=1M
1144+0 records in
1144+0 records out
1199570944 bytes (1,2 GB, 1,1 GiB) copied, 214,305 s, 5,6 MB/s
$sudo sync

    Let's boot it and perform initial configuration, like account creation, setting up passwords and whatever else you need to claim it ready for taking snapshot

    I'll start screen to monitor boot process via USB-UART adapter by issuing command

    $ screen /dev/ttyUSB0 115200

     Result is below

    U-Boot SPL 2021.04-armbian (Jun 21 2021 - 15:06:33 +0000)
DRAM: 256 MiB
Trying to boot from MMC1


U-Boot 2021.04-armbian (Jun 21 2021 - 15:06:33 +0000) Allwinner Technology

CPU:   Allwinner H3 (SUN8I 1680)
Model: Xunlong Orange Pi Zero
DRAM:  256 MiB
MMC:   mmc@1c0f000: 0, mmc@1c10000: 1
Loading Environment from FAT... Unable to use mmc 0:1... In:    serial
Out:   serial
Err:   serial
Net:   phy interface0
eth0: ethernet@1c30000
starting USB...

[...]

[  OK  ] Started Network Manager Script Dispatcher Service.
[  OK  ] Started Dispatcher daemon for systemd-networkd.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Finished Update UTMP about System Runlevel Changes.

orangepizero login: root (automatic login)

  ___  ____  _   _____              
 / _ \|  _ \(_) |__  /___ _ __ ___  
| | | | |_) | |   / // _ \ '__/ _ \ 
| |_| |  __/| |  / /|  __/ | | (_) |
 \___/|_|   |_| /____\___|_|  \___/ 
                                    
Welcome to Armbian 21.05.6 Focal with Linux 5.10.43-sunxi

System load:   100%             Up time:       2 min
Memory usage:  47% of 238M      IP:            
CPU temp:      54°C             Usage of /:    8% of 15G    
RX today:      n/a        

[ General system configuration (beta): armbian-config ]

New to Armbian? Documentation: https://docs.armbian.com Support: https://forum.armbian.com

New root password:

    Next step is to enable SPI flash and flash it with u-boot

  • Introduction

    andriy.malyshenko • 08/11/2021 at 19:38 • 0 comments

    Recently I was playing around Orange Pi Zero (actually checking if I'd be able to boot it without any storage). Almost accidentally I found out that recent u-boot able to use onboard Ethernet for boot purposes. Checked everything and it did indeed worked. I want to note my steps for anyone willing to repeat that process.


    First of all prerequisites:

    - Orange Pi Zero - tutorial would not work for just any Orange Pi out there. I'm using onboard SPI flash to start boot process, and per my knowledge only Orange Pi Zero has both SPI flash and Ethernet PHY.

    - To configure Orange Pi we need SD card. One finished it will be no longer required. It is not strictly necessary, but makes things easier.

    - (optional) To debug whole process it is really helpful to have USB-UART adapter. It will cost you around $1 if you still don't have one. It is not strictly necessary, but will help a lot in understanding the process

    From your network perspective

    - (A) Some router serving as DHCP server. When Pi will bot it will need clear instruction where boot components should be taken from.

    - (B) Some network server with storage, preferably fast one. It will store Pi filesystems

    - (C) Some network server with boot artifacts exposed via TFTP server.

    All of the above can be combined in single physical machine, or split roles in every possible combination. 

    From the perspective of going through this tutorial we would need some wokrplace PC or laptop with access to the same network. This is where we're sitting and deploying all the magic.

    The process in general will look like this

    1. We prepare standard bootable image on SD card and boot it on our Pi. We will need it to (a) flash u-boot to onboard SPI flash and (b) to take filesystem snapshot

    2. We will configure NFS (network accessible filesystem) and take snapshot from running system.

    3. We will configure TFTP service on network boot server and configure DHCP to instruct our Pi to use it.

    Boot process in short: