netmap on cxgbe interfaces

Last year my testbed machine got the luxury of 100Gbit Chelsio T6 network interfaces with a pair of T62100-LP-CR cards. I have been using these recently for VPP development on FreeBSD and while they work well I have been having a lot of trouble doing any packet generation with them.

I got pretty miserable performance using netmaps pkt-gen on FreeBSD and trying Linux the comparative oddness of the T6 drivers seems to trip on the TRex packet generator suite on Linux.

From reading Chelsio's own documentation it seemed I was missing something, they have benchmarks using pkt-gen showing link saturation for small packets at 40Gbit/s on their T5 cards. What was up?

When I run a pkt-gen (lives in tools/tools/netmap ) benchmark I get a little under 1 Gbit/s of traffic:

$ sudo ./pkt-gen -f tx -i cc0 -S 00:07:43:74:3f:e9 -D 00:07:43:74:3f:e1
...
250.554583 main_thread [2713] 9.012 Mpps (9.587 Mpkts 4.326 Gbps in 1063799 usec) 512.00 avg_batch 0 min_space

~9 Mpps isn't that great, I paid for 100 Gbit and 4 just isn't enough.

FreeBSD has a zero copy userspace networking framework called netmap , it cleverly hooks into drivers early in their packet processing pipeline and if there is a netmap application running steals them so the host doesn't see them. Aiding with support, if the app doesn't want the given packet it can be given back to the driver and normal processing can occur.

Drivers can be natively supported with netmap or if support isn't available an emulated mode driver can be used.

Eventually after looking really really hard at the Chelsio examples I saw that the network interface names were different from mine. They were prefixed with a 'v', 'vcxl' rather than 'cxl'.

A bunch of googling later I read the top cxgbe(4) correctly:

The cxgbe driver uses different names for devices based on the associated
ASIC:

      ASIC    Port Name    Parent Device    Virtual Interface
      T4      cxgbe        t4nex            vcxgbe
      T5      cxl          t5nex            vcxl
      T6      cc           t6nex            vcc

To get a virtual interface we need to set the hw.cxgbe.num_vis sysctl to a value greater than 1. I added the following to my /boot/loader.conf :

# Add a virtual port for cxgbe                                     
hw.cxgbe.num_vis="2"            # Number of VIs per port           
hw.cxgbe.nnmrxq_vi="8"          # Number of netmap RX queues per VI
hw.cxgbe.nnmtxq_vi="8"          # Number of netmap TX queues per VI
hw.cxgbe.nrxq_vi="8"            # Number of RX queues per VI       
hw.cxgbe.ntxq_vi="8"            # Number of TX queues per VI

On a reboot I have a shiny new vcc0 device, letting pkt-gen run there I get much nicer results:

$ sudo ./pkt-gen -f tx -i vcc0 -S 00:07:43:74:3f:e9 -D 00:07:43:74:3f:e1 
...
329.706767 main_thread [2713] 68.770 Mpps (73.060 Mpkts 33.010 Gbps in 1062380 usec) 480.61 avg_batch 99999 min_space

~70 Mpps and 33 Gbps of 64 byte packets is pretty great , with an imix I can saturate the 100Gbps with pkt-gen .

I didn't find anything about this differentiator by searching, pkt-gen doesn't mention it. There are some dmesg lines, but if you don't know, you don't know.

238.184192 [1135] generic_netmap_attach     Emulated adapter for cc0 created (prev was NULL)
238.192821 [ 320] generic_netmap_register   Emulated adapter for cc0 activated
238.200244 [1889] netmap_interp_ringid      invalid ring id 1
238.206046 [ 295] generic_netmap_unregister Emulated adapter for cc0 deactivated

Booting Linux on the Pine64 Ox64 SBC

The Ox64 (notice that is a letter 'O' not a 0) is a tiny microcontroller sized single board computer from Pine64. The Ox64 is built around the bl808 SOC from Bouffalo Lab. It has one application level 64 bit RISC-V core, a 32 bit RISC-V core and an extra little 32 bit RISC-V core for handling io.

It is a computer in the form factor of a microcontroller and it is built around what seems to be an open eco system. The Ox64 is early in its lifetime and Pine64 are quite good at making hardware for the community to build software around. That means that software is early and there is much to do (as I learned) to get code running.

Lets get Bouffalo's example Linux build going on this tiny thing.

First signs of life

There is quite a bit of documentation and write ups in the wiki page which helped me get going.

The links in the wiki aren't super clear about how to get any sort of working Linux image on the Ox64. Before trying any of those I wanted to get some signs of life out of the board. A two hour round trip to the hackerspace later I had headers soldered on to my two boards and connected up the Waveshare USB uart I got with the VisionFive2 kit.

The instructions aren't clear about where to expect the initial uart output to appear. The pinout on the wiki has RXD and TXD marked in several places, but also GPIO14/GPIO15 for UART0 TX adn UART0 RX. Two baud rates are listed 115200 and 2000000 (this might be faster than your usb uart can go), I got the following out of GPIO14/15 with a baud rate of 2000000.

$ sudo cu -l /dev/ttyU0 -s 2000000

 ____               __  __      _       _       _     
|  _ \             / _|/ _|    | |     | |     | |    
| |_) | ___  _   _| |_| |_ __ _| | ___ | | __ _| |__  
|  _ < / _ \| | | |  _|  _/ _` | |/ _ \| |/ _` | '_ \ 
| |_) | (_) | |_| | | | || (_| | | (_) | | (_| | |_) |
|____/ \___/ \__,_|_| |_| \__,_|_|\___/|_|\__,_|_.__/ 

Build:19:50:39,Nov 20 2022
Copyright (c) 2022 Bouffalolab team
dynamic memory init success,heap size = 93 Kbyte 
sig1:ffff32ff
sig2:0000ffff
Pong!
Ping!
Pong!
Ping!
Pong!

Flashing with DevCube

The buildroot instructions on the wiki have an example usage of the SDK to build firmware and how to flash boot firmware for the two cores.

Bouffalo's flashing tool is called Devcube, they provide binaries for Linux, Mac OS and Windows. The Linux Binary (version 1.8.4) refused to run on FreeBSD with Linux compat giving me:

$ ./BLDevCube-ubuntu 
PySide2/__init__.py: Unable to import shiboken2 from /tmp/_MEIN3lvWA/base_library.zip, /tmp/_MEIN3lvWA/lib-dynload, /tmp/_MEIN3lvWA
/lib64/libc.so.6: version `GLIBC_2.18' not found (required by /tmp/_MEIN3lvWA/libstdc++.so.6)

I don't have a lot of experience using Linux compat and I want to get this board going in less than a month of work so I flashed from Mac OS.

In Mac OS I was able to run Devcube, but kept hitting failures trying to flash the firmware images onto the board. Eventually I started trying more extreme things, the wiki page suggested that certain USB uart chips just don't work, so I tried another adapter. Another suggested solution was to write a USB uart firmware on to a pi pico and use that. I pulled the pi pico from my BluesSCSI v2, copied the firmware on to the virtual msdos file system and that finally worked.

With the firmware written the output on uart0 changed to:

[I][] 
[I][]   ____                   ____               __  __      _       
[I][]  / __ \                 |  _ \             / _|/ _|    | |      
[I][] | |  | |_ __   ___ _ __ | |_) | ___  _   _| |_| |_ __ _| | ___  
[I][] | |  | | '_ \ / _ \ '_ \|  _ < / _ \| | | |  _|  _/ _` | |/ _ \ 
[I][] | |__| | |_) |  __/ | | | |_) | (_) | |_| | | | || (_| | | (_) |
[I][]  \____/| .__/ \___|_| |_|____/ \___/ \__,_|_| |_| \__,_|_|\___/ 
[I][]        | |                                                      
[I][]        |_|                                                      
[I][] 
[I][] Powered by BouffaloLab
[I][] Build:11:52:22,Mar  6 2023
[I][] Copyright (c) 2023 OpenBouffalo team
[I][] Copyright (c) 2022 Bouffalolab team
[I][] =========== flash cfg ==============
[I][] jedec id   0xEF6018
[I][] mid            0xEF
[I][] iomode         0x04
[I][] clk delay      0x01
[I][] clk invert     0x01
[I][] read reg cmd0  0x05
[I][] read reg cmd1  0x35
[I][] write reg cmd0 0x01
[I][] write reg cmd1 0x31
[I][] qe write len   0x01
[I][] cread support  0x00
[I][] cread code     0xFF
[I][] burst wrap cmd 0x77
[I][] sector size:   0x04
[I][] =====================================
[I][] dynamic memory init success,heap size = 156 Kbyte 
[I][MAIN] Starting Mailbox Handlers
[I][MBOX] Forwarding Interupt SDH (33) to D0 (0x58008bbc)
[I][MBOX] Forwarding Interupt GPIO (60) to D0 (0x58008d0e)
[I][MAIN] Running...
[I][MBOX] Mailbox IRQ Stats:
[I][MBOX]   Peripheral SDH (33): 0
[I][MBOX]   Peripheral GPIO (60): 0
[I][MBOX] Unhandled Interupts: 0 Unhandled Signals 0
[I][MBOX] ====================================
[I][MBOX] Mailbox IRQ Stats:
[I][MBOX]   Peripheral SDH (33): 0
[I][MBOX]   Peripheral GPIO (60): 0
[I][MBOX] Unhandled Interupts: 0 Unhandled Signals 0
[I][MBOX] ====================================
[I][MBOX] Mailbox IRQ Stats:
[I][MBOX]   Peripheral SDH (33): 0
[I][MBOX]   Peripheral GPIO (60): 0
[I][MBOX] Unhandled Interupts: 0 Unhandled Signals 0
[I][MBOX] ====================================

This is promising and nice to see firmware giving us a channel for further debugging in the future.

I wrote the example sd card image onto a card and tried to boot and got nothing.

In the instructions I spotted that the uart for u-boot/linux was on gpio 16/17 on the other side of the board to where I had soldered headers (I only had enough right angled headers to do one side). Luckily, I soldered straight headers onto all pins of the second Ox64 I got for working on this.

With those connected I still got nothing, I tried all the uarts on the pinout in the wiki. Dropping to 115200 for the baud because some disconnected messages in the discord suggested that might be the issue. I verified that the baud should be 2000000 in the DTB files.

I gave in, joined the Pine64 discord and asked what might be up. Arne said that Devcube 1.8.4 has some konwn issues and I should try Devcube 1.8.3 . This version didn't like uploading the bl808-firmware.bin file, it was unable to establish a connection to the firmware and gave me the same errors I saw before when I was using the USB uart adapter.

I tried fruitlessly many combinations of boot switches and power cycles, but I couldn't get the firmware to write.

Not sure what to try next I connected up to the system uart to see if I could get any further hints and instead I got a booting Linux:

[I][] 
[I][]   ____                   ____               __  __      _       
[I][]  / __ \                 |  _ \             / _|/ _|    | |      
[I][] | |  | |_ __   ___ _ __ | |_) | ___  _   _| |_| |_ __ _| | ___  
[I][] | |  | | '_ \ / _ \ '_ \|  _ < / _ \| | | |  _|  _/ _` | |/ _ \ 
[I][] | |__| | |_) |  __/ | | | |_) | (_) | |_| | | | || (_| | | (_) |
[I][]  \____/| .__/ \___|_| |_|____/ \___/ \__,_|_| |_| \__,_|_|\___/ 
[I][]        | |                                                      
[I][]        |_|                                                      
[I][] 
[I][] Powered by BouffaloLab
[I][] Build:11:52:04,Mar  6 2023
[I][] Copyright (c) 2023 OpenBouffalo team
[I][] Copyright (c) 2022 Bouffalolab team
[I][] dynamic memory init success,heap s[I][LowLoad] D0 start...
[I][LowLoad] low_load start... 
[I][LowLoad] Header at 0x5d5ff000
[I][LowLoad] Section dtb(1) - Start 0x5d5ff100, Size 14314
[I][LowLoad] Copying DTB to 0x51ff8000...0x51ffb7ea
[I][LowLoad] Done!
[I][LowLoad] Section OpenSBI(2) - Start 0x5d60f100, Size 109864
[I][LowLoad] Copying OpenSBI to 0x3ef80000...0x3ef9ad28
[I][LowLoad] Done!
[I][LowLoad] Section Kernel(3) - Start 0x5d62f100, Size 315597
[I][LowLoad] Uncompressing Kernel to 0x50000000...
[I][LowLoad] Done!
[I][LowLoad] CRC: 00000000
[I][LowLoad] load time: 61311 us 
[I][LowLoad] Setting PMP
[I][LowLoad] Booting OpenSBI at 0x000000003ef80000 with DTB at 0x51ff8000

OpenSBI v1.2
  ____                    _____ ____ _____
 / __ \                  / ____|  _ \_   _|
| |  | |_ __   ___ _ __ | (___ | |_) || |
| |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
| |__| | |_) |  __/ | | |____) | |_) || |_
 \____/| .__/ \___|_| |_|_____/|____/_____|
       | |
       |_|

  Platform Name             : Pine64 Ox64 (D0)
  Platform Features         : medeleg
  Platform HART Count       : 1
  Platform IPI Device       : aclint-mswi
  Platform Timer Device     : aclint-mtimer @ 1000000Hz
  Platform Console Device   : bflb_uart
  Platform HSM Device       : ---
  Platform PMU Device       : ---
  Platform Reboot Device    : ---
  Platform Shutdown Device  : ---
  Firmware Base             : 0x3ef80000
  Firmware Size             : 200 KB
  Runtime SBI Version       : 1.0

  Domain0 Name              : root
  Domain0 Boot HART         : 0
  Domain0 HARTs             : 0*
  Domain0 Region00          : 0x00000000e4008000-0x00000000e400bfff (I)
  Domain0 Region01          : 0x00000000e4000000-0x00000000e4007fff (I)
  Domain0 Region02          : 0x000000003ef80000-0x000000003efbffff ()
  Domain0 Region03          : 0x0000000000000000-0xffffffffffffffff (R,W,X)
  Domain0 Next Address      : 0x0000000050000000
  Domain0 Next Arg1         : 0x0000000051ff8000
  Domain0 Next Mode         : S-mode
  Domain0 SysReset          : yes

  Boot HART ID              : 0
  Boot HART Domain          : root
  Boot HART Priv Version    : v1.11
  Boot HART Base ISA        : rv64imafdcvx
  Boot HART ISA Extensions  : time
  Boot HART PMP Count       : 8
  Boot HART PMP Granularity : 4096
  Boot HART PMP Address Bits: 38
  Boot HART MHPM Count      : 8
  Boot HART MIDELEG         : 0x0000000000000222
  Boot HART MEDELEG         : 0x000000000000b109


  U-Boot 2023.04-rc2 (Mar 06 2023 - 11:48:40 +0000)

  DRAM:  64 MiB
  Core:  36 devices, 17 uclasses, devicetree: board
  MMC:   mmc@20060000: 0
  Loading Environment from FAT... Unable to read "uboot.env" from mmc0:2... 
  Loading Environment from nowhere... OK
  In:    serial@30002000
  Out:   serial@30002000
  Err:   serial@30002000
  Net:   
  Warning: emac@20070000 (eth0) using random MAC address - fa:6f:53:9f:5c:36
  eth0: emac@20070000
  Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:2...
Found /extlinux/extlinux.conf
Retrieving file: /extlinux/extlinux.conf
Select the boot mode
1:  Pine64 0X64 Kernel
2:  Sipeed M1SDock Kernel
Enter choice: 1
1:  Pine64 0X64 Kernel
Retrieving file: /extlinux/../Image
append: root=PARTLABEL=rootfs rootwait rw rootfstype=ext4 console=ttyS0,2000000 loglevel=8 earlycon=sbi
Retrieving file: /extlinux/../bl808-pine64-ox64.dtb
## Flattened Device Tree blob at 51ff8000
Booting using the fdt blob at 0x51ff8000
Working FDT set to 51ff8000
Loading Device Tree to 0000000053f22000, end 0000000053f25fab ... OK
Working FDT set to 53f22000

Starting kernel ...

[    0.000000] Linux version 6.2.0 (runner@fv-az587-938) (riscv64-unknown-linux-gnu-gcc (Xuantie-900 linux-5.10.4 glibc gcc Toolchain V2.6.1 B-20220906) 10.2.0, GNU ld (GNU Binutils) 2.35) #1 Mon Mar  6 11:17:27 UTC 2023
[    0.000000] OF: fdt: Ignoring memory range 0x50000000 - 0x50200000
[    0.000000] Machine model: Pine64 Ox64
[    0.000000] earlycon: sbi0 at I/O port 0x0 (options '')
[    0.000000] printk: bootconsole [sbi0] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000050200000-0x0000000053ffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000050200000-0x0000000053ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000050200000-0x0000000053ffffff]
[    0.000000] SBI specification v1.0 detected
[    0.000000] SBI implementation ID=0x1 Version=0x10002
[    0.000000] SBI TIME extension detected
[    0.000000] SBI IPI extension detected
[    0.000000] SBI RFENCE extension detected
[    0.000000] riscv: base ISA extensions acdfim
[    0.000000] riscv: ELF capabilities acdfim
[    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[    0.000000] pcpu-alloc: [0] 0 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 15655
[    0.000000] Kernel command line: root=PARTLABEL=rootfs rootwait rw rootfstype=ext4 console=ttyS0,2000000 loglevel=8 earlycon=sbi
[    0.000000] Dentry cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.000000] Inode-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 48084K/63488K available (3941K kernel code, 4584K rwdata, 2048K rodata, 2118K init, 301K bss, 15404K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] riscv-intc: 64 local interrupts mapped
[    0.000000] plic: interrupt-controller@e0000000: mapped 64 interrupts with 1 handlers for 2 contexts.
[    0.000000] riscv-timer: riscv_timer_init_dt: Registering clocksource cpuid [0] hartid [0]
[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0x1d854df40, max_idle_ns: 3526361616960 ns
[    0.000003] sched_clock: 64 bits at 1000kHz, resolution 1000ns, wraps every 2199023255500ns
[    0.000920] Console: colour dummy device 80x25
[    0.001166] Calibrating delay loop (skipped), value calculated using timer frequency.. 2.00 BogoMIPS (lpj=4000)
[    0.001687] pid_max: default: 32768 minimum: 301
[    0.002408] Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.002749] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.007710] cblist_init_generic: Setting adjustable number of callback queues.
[    0.008028] cblist_init_generic: Setting shift to 0 and lim to 1.
[    0.008940] ASID allocator using 16 bits (65536 entries)
[    0.010160] EFI services will not be available.
[    0.011191] devtmpfs: initialized
[    0.014879] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.015375] futex hash table entries: 256 (order: 0, 6144 bytes, linear)
[    0.016003] pinctrl core: initialized pinctrl subsystem
[    0.018536] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.019435] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
[    0.019823] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.025441] bflb-ipc 30005000.mailbox: Bouffalo Lab IPC mailbox interrupt controller
[    0.030626] SCSI subsystem initialized
[    0.033265] clocksource: Switched to clocksource riscv_clocksource
[    0.061833] NET: Registered PF_INET protocol family
[    0.062497] IP idents hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.064593] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.065029] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.065544] TCP established hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.065953] TCP bind hash table entries: 512 (order: 1, 8192 bytes, linear)
[    0.066387] TCP: Hash tables configured (established 512 bind 512)
[    0.067071] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.067407] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.068175] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.070709] workingset: timestamp_bits=62 max_order=14 bucket_order=0
[    0.073019] NET: Registered PF_ALG protocol family
[    0.073455] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[    0.073801] io scheduler mq-deadline registered
[    0.074056] io scheduler kyber registered
[    0.074387] io scheduler bfq registered
[    0.075570] bflb-gpio-pinctrl 200008c4.pinctrl: No cache defaults, reading back from HW
[    0.077954] bflb-gpio-pinctrl 200008c4.pinctrl: Bouffalo Lab pinctrl+GPIO(+interrupt) controller - Registered 32 function(s) for 46 pin(s)
[    0.083806] 30002000.serial: ttyS0 at MMIO 0x30002000 (irq = 2, base_baud = 2500000) is a BFLB UART
[    0.084288] printk: console [ttyS0] enabled
[    0.084288] printk: console [ttyS0] enabled
[    0.084735] printk: bootconsole [sbi0] disabled
[    0.084735] printk: bootconsole [sbi0] disabled
[    0.086406] 2000aa00.serial: ttyS1 at MMIO 0x2000aa00 (irq = 3, base_baud = 2500000) is a BFLB UART
[    0.115172] brd: module loaded
[    0.131893] loop: module loaded
[    0.133183] physmap-flash 58500000.xip_flash: physmap platform flash device: [mem 0x58500000-0x588fffff]
[    0.139081] wireguard: WireGuard 1.0.0 loaded. See www.wireguard.com for information.
[    0.139577] wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
[    0.140210] PPP generic driver version 2.4.2
[    0.140910] PPP BSD Compression module registered
[    0.141223] PPP Deflate Compression module registered
[    0.142332] sdhci: Secure Digital Host Controller Interface driver
[    0.142723] sdhci: Copyright(c) Pierre Ossman
[    0.142993] sdhci-pltfm: SDHCI platform and OF driver helper
[    0.144330] mmc0 bounce up to 128 segments into one, max segment size 65536 bytes
[    0.145494] ledtrig-cpu: registered to indicate activity on CPUs
[    0.146299] bflb-seceng 20004000.seceng: No cache defaults, reading back from HW
[    0.149836] random: crng init done
[    0.150084] bflb-seceng 20004000.seceng: Bouffalo Lab Secure Engine
[    0.151045] riscv-pmu-sbi: SBI PMU extension is available
[    0.151453] riscv-pmu-sbi: 16 firmware and 10 hardware counters
[    0.153359] NET: Registered PF_INET6 protocol family
[    0.156148] Segment Routing with IPv6
[    0.156517] In-situ OAM (IOAM) with IPv6
[    0.156953] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    0.158856] NET: Registered PF_PACKET protocol family
[    0.159229] Key type dns_resolver registered
[    0.183293] debug_vm_pgtable: [debug_vm_pgtable         ]: Validating architecture page table helpers
[    0.197399] mmc0: SDHCI controller on 20060000.sdhci [20060000.sdhci] using DMA
[    0.198161] Waiting for root device PARTLABEL=rootfs...
[    0.489067] mmc0: new high speed SDHC card at address b368
[    0.491548] mmcblk0: mmc0:b368 NCard 29.1 GiB 
[    0.510192]  mmcblk0: p1 p2 p3
[    0.551381] EXT4-fs (mmcblk0p3): mounted filesystem c0eebeb6-58dd-4946-bc48-e8c2c43098cc with ordered data mode. Quota mode: disabled.
[    0.552227] VFS: Mounted root (ext4 filesystem) on device 179:3.
[    0.559988] devtmpfs: mounted
[    0.568251] Freeing unused kernel image (initmem) memory: 2116K
[    0.568740] Run /sbin/init as init process
[    0.569018]   with arguments:
[    0.569212]     /sbin/init
[    0.569447]   with environment:
[    0.569654]     HOME=/
[    0.569812]     TERM=linux
[    0.841528] EXT4-fs (mmcblk0p3): re-mounted c0eebeb6-58dd-4946-bc48-e8c2c43098cc. Quota mode: disabled.
[    1.647489] Adding 1048572k swap on /dev/mmcblk0p1.  Priority:-2 extents:1 across:1048572k SS
Root Partition Already resized
Starting syslogd: OK
Starting klogd: OK
Running sysctl: OK
Starting mdev... OK
Initializing random number generator: OK
Saving random seed: OK
Starting rpcbind: OK
Starting iptables: OK
Starting network: OK
Starting dhcpcd...
dhcpcd-9.4.1 starting
dhcp_vendor: Invalid argument
forked to background, child pid 103
Starting sntp: sntp 4.2.8p15@1.3728-o Mon Mar  6 10:45:15 UTC 2023 (1)
pool.ntp.org lookup error Temporary failure in name resolution
FAIL
Starting ntpd: OK
Starting dropbear sshd: OK

Welcome to Buildroot
ox64 login:

Cool, that is Linux going, with only a lot of hassle. I wonder what I could try next?

BSDCan 2023

In the Days before my fiancée and I explored Ottawa a bit, got stranded in a forest in Quebec and did our usual tour of coffee shops and book stores (got to make sure your bags are heavier on the way home somehow).

The evening before BSDCan and the Devsummits is traditionally a meeting of the Editorial Board of the FreeBSD Journal and accomplices to discuss what we want to cover in the next year. We had a brain storming session that picked up nearly 40 ideas for articles. From this we will figure out topics for the issues in the next year.

Devsummit Day 1

Typically in the days before a BSD conference the FreeBSD project holds a developer summit with some time aside for hacking . BSDCan is a big event for this and the Devsummit before was well attended.

Foundation update

The Devsummit normally starts with an update from the FreeBSD foundation on what is going on. This summits update included a lot of information about on going projects, new hires and direction the foundation is working in. After a pandemic gap in major updates there was a lot of new information from the foundation.

This year is the 30th anniversary of the start of the FreeBSD project and the foundation got us cake to celebrate!

Core update

Next up is normally an update from core letting us know what they are doing about it.

Guided code reading with Mark J.

For a new Devsummit thing Mark Johnson did a guided code reading session in the scheduler. This was an introduction to the files and functions of the ULE scheduler ifreebsd which has been a topic of discussion in recent years and more in the last two months.

I think as a new format this worked quite well, taking 40 minutes or so to walk around a sun system might be a good way to give devs kick start introductions when help is needed.

Story time with Mike Karels

Wrapping up the first Devsummit day we had a story time session with Mike Karels. Mike is one of the OG BSD people, it was great to hear his experiences working with and on BSD for the last 4 decades. This one was recorded and will be worth revisiting in the future.

Devsummit Day 2

I took the morning of the second day of the Devsummit to stare into my slides and wonder why I had signed up to talk again. For me this is normal fair in the lead up to a conference, but boy am I good at forgetting about all the lead in anxiety before presenting a talk.

15 planning

I made it to the Devsummit to join the have/need/want/axe session for FreeBSD 15. These sessions are where we try to figure out what we might have for an upcoming release and what we need to get devs to sign up to do.

In the last few years we've had more desire to kill off old subsystems and tools, making the code base both smaller and easier to maintain. 15 will continue this and there is a nice list of things to get rid of.

Group photo and off to the pub to help set up for registration

Post lunch we had a project group photo of everyone that made the Devsummit. I decided to duck out for the rest of the day and help Dan prepare stuff for registration for start of the conference. It helped that this prep was in a pub.

BSDCan Day 1

I took the morning of the first day to review my slides, drink coffee and try to avoid becoming agitated before my talk skipping the two sessions which happened in the slots before.

My talk - Making FreeBSD QUIC

I reprised my talk from EuroBSDCon, I had hoped to expand on it, but procrastination and time limitations got in the way. I’m working on a hacky GSO for FreeBSD, while it would have been great to present early results, they just weren't ready in time.

I think the talk went well, but as it was a second presentation that isn’t too surprising. If I present at a BSD conference again it’ll be something completely new and weirdly out of the box.

Mateusz - Dtrace vs eBPF

An update from EuroBSDCon 2022, Mateusz looked at the tracing overheads for eBPF and DTrace . There are a lot of claims out there about how things work and the impact performance tools have ranging from no overhead so the tool is safe to use all the way up to too much overhead so tools are never safe to use.

Mateusz normalised out differences between FreeBSD and Linux and explored the overheads of both sets of tools on each OS.

Kirk - gunion

Kirk came to give an overview of a new geom module call gunion:

“Turns out explains gunion takes about 10 minutes so I’ll give some background on geom too”

Kirk covered why we have geom and how it developed over time. He talked about the nop geom which has grown enough features to not be a no operation anymore. He wrapped up talking about gunion - a new geom writable layer that grabs writes to a paired geom allowing you to dry run potentially destructive operations. The use came for this is a quick fsck -y without the pitfall fall of potentially killing your disk.

Me again, with Benedict and Allan - BSDNow Live

Final session for day 1 a live performance of BSDNow. We went into this one with as much prep as we normally do and that just wasn't enough.

Our big announcement at the end is that Allan is going to retire from hosting the show with the live show being his final episode.

I got positive feedback after and I may be tempted to try this format again. The recording is the record of if this was a good use of everyone’s time.

In the future I'll ask to have the hour before the live show blocked out for set up and preparation and bring enough of a mic set up that we can be heard in the room, on the stream and have a good mix for the show. This 'speak into the goat' experience while very funny was not a great recording environment.

BSDCan Day 2

I started the second day with a detour to get good coffee from Little Victories and got caught in the start of the day of rain. The little victories chain (there are 2 in 2023) is my recommendation for the best coffee near the venue, I made the 10 minute detour twice on my way in to have “good” coffee.

Kristof - if_ovpn

The first session was Kristof talking about the design and implementation of the if_ovpn kernel module. This modules allows crypto and packet processing to be handled in the kernel avoiding a round trip to user space for every packet.

This works in a similar way to TLS offload where the meat is handled in the kernel, but the slower control traffic has to dealt with my a user space process.

It seemed to offer pretty good performance improvements over the user space side; almost 5x

Taylor Campbell - NetBSD disk encryption

It is always good to know what is going on in other parts of the BSD world and with two competing talks which I’d watch on YouTube I decided to see a NetBSD talk.

Taylor spoke about the design and usage of CGD disk device which offers encryption in NetBSD. He talked about some of the details and limitations of the current system. I’m glad that there are modern descriptions of NetBSD kernel internals happening at conferences. I do wish I’d managed to see one of the OpenBSD sessions but it didn't work out.

Corey Stephan - Using BSD in the liberal arts

Corey’s talk was a change of pace from the normal content at a BSD conference. Corey is a theologian teaching at a Texan university.

“Who has sufficient time to build a better computer workflow”

He spoke about the role open software and our communities can have in the study and teaching of liberal arts. Corey has had good experiences working with us BSD folk and is able to work well with packaged software. I’m hoping he’ll give an update in a few years and tell us if he managed to convert anyone else.

Lunch interview with the freebsd foundation

During lunch I did a short “how you got into bsd” interview with Drew from the foundation. I think it’s going into TikTok or something similar.

Wuyang Chung - single address space capability system

Another change of pace and a very academic style presentation.

Warner Losh - kboot: Booting FreeBSD with LinuxBoot

LinuxBoot offers a mechanism for a Linux kernel to boot another kernel, this can have a huge advantage if you need to reboot something that takes a long time to move through firmware initialisation. With LinuxBoot you kexec the new image and can do a warm restart.

Warner did work to enable FreeBSD to be booted with kexec/Linuxboot. This is similar to the way FreeBSD could be booted on the PlayStation 3, but with a decade plus of new stuff added in.

Closing and Auction

BSDCan ends each year with a closing ceremony and auction. In the closing ceremony Dan announced that he was stepping down after 20 years of running the conference. New team is coming together to keep BSDCan going into its third decade.

The final event is an auction to raise money for the Ottawa mission. I joined in and picked up a 5th copy of the 30th anniversary journal edition with signatures from the people at the conference.

Booting FreeBSD on the RISC-V VisionFive2

The VisionFive 2 is a RISC-V based single board computer from Starfive integrating their JH7110 SoC. The board is available from Waveshare on its own or with some handy components to get you started.

The SoC is pitched to be a competitor to the RaspberryPi 4 in specs and along with oodles of interfaces it has a raspberrypi compatible 40 pin header exposing pins for io.

The SoC is also in the Star64 and PinetabV from Pine64 (I didn't know that the StarFive was for sale until checking when writing this).

StarFive is a Chinese integrator using IP blocks from SiFive and so it integrates cores and peripherals from the HiFive Unleashed board.

I picked one of these up to play with FreeBSD/RISC-V on a computer with a real MMU. Before doing anything I wanted to be sure that I could get Linux to boot on the board. I sort of followed the instructions from the [Waveshare wiki page][wavesharewiki], ignoring the Windows GUI tools and using dd and cu to write the image and connect to serial.

With an SD card image with their published I powered up the board and tried to connect. The wiki instructions highlight that you need to break into the boot loader during boot and manually load segments from the SD card. I tried this for about an hour or two with 3 different images (the 69 image took 40 minutes to write to the SD card), but I didn't have any luck.

Giving up I just let the u-boot run and got a booting Debian system immediately:

U-Boot 2021.10 (Feb 12 2023 - 18:15:33 +0800), Build: jenkins-VF2_515_Branch_SDK_Release-24

CPU:   rv64imacu
Model: StarFive VisionFive V2
DRAM:  8 GiB
MMC:   sdio0@16010000: 0, sdio1@16020000: 1
Loading Environment from SPIFlash... SF: Detected gd25lq128 with page size 256 Bytes, erase size 4 KiB, total 16 MiB
*** Warning - bad CRC, using default environment

StarFive EEPROM format v2

--------EEPROM INFO--------
Vendor : StarFive Technology Co., Ltd.
Product full SN: VF7110B1-2253-D008E000-00004015
data version: 0x2
PCB revision: 0xb2
BOM revision: A
Ethernet MAC0 address: 6c:cf:39:00:42:ba
Ethernet MAC1 address: 6c:cf:39:00:42:bb
--------EEPROM INFO--------

In:    serial@10000000
Out:   serial@10000000
Err:   serial@10000000
Model: StarFive VisionFive V2
Net:   eth0: ethernet@16030000, eth1: ethernet@16040000
switch to partitions #0, OK
mmc1 is current device
found device 1
bootmode flash device 1
385 bytes read in 6 ms (62.5 KiB/s)
Importing environment from mmc1 ...
Can't set block device
Hit any key to stop autoboot:  0
Can't set block device
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Retrieving file: /boot/extlinux/extlinux.conf
Can't set block device
Error reading config file
StarFive #
StarFive #
U-Boot SPL 2021.10 (Feb 12 2023 - 18:15:33 +0800)
DDR version: dc2e84f0.
Trying to boot from SPI

OpenSBI v1.2
   ____                    _____ ____ _____
  / __ \                  / ____|  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) |  __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|____/_____|
        | |
        |_|

Platform Name             : StarFive VisionFive V2
Platform Features         : medeleg
Platform HART Count       : 5
Platform IPI Device       : aclint-mswi
Platform Timer Device     : aclint-mtimer @ 4000000Hz
Platform Console Device   : uart8250
Platform HSM Device       : jh7110-hsm
Platform PMU Device       : ---
Platform Reboot Device    : pm-reset
Platform Shutdown Device  : pm-reset
Firmware Base             : 0x40000000
Firmware Size             : 292 KB
Runtime SBI Version       : 1.0

Domain0 Name              : root
Domain0 Boot HART         : 1
Domain0 HARTs             : 0*,1*,2*,3*,4*
Domain0 Region00          : 0x0000000002000000-0x000000000200ffff (I)
Domain0 Region01          : 0x0000000040000000-0x000000004007ffff ()
Domain0 Region02          : 0x0000000000000000-0xffffffffffffffff (R,W,X)
Domain0 Next Address      : 0x0000000040200000
Domain0 Next Arg1         : 0x0000000042200000
Domain0 Next Mode         : S-mode
Domain0 SysReset          : yes

Boot HART ID              : 1
Boot HART Domain          : root
Boot HART Priv Version    : v1.11
Boot HART Base ISA        : rv64imafdcbx
Boot HART ISA Extensions  : none
Boot HART PMP Count       : 8
Boot HART PMP Granularity : 4096
Boot HART PMP Address Bits: 34
Boot HART MHPM Count      : 2
Boot HART MIDELEG         : 0x0000000000000222
Boot HART MEDELEG         : 0x000000000000b109


U-Boot 2021.10 (Feb 12 2023 - 18:15:33 +0800), Build: jenkins-VF2_515_Branch_SDK_Release-24

CPU:   rv64imacu
Model: StarFive VisionFive V2
DRAM:  8 GiB
MMC:   sdio0@16010000: 0, sdio1@16020000: 1
Loading Environment from SPIFlash... SF: Detected gd25lq128 with page size 256 Bytes, erase size 4 KiB, total 16 MiB
*** Warning - bad CRC, using default environment

StarFive EEPROM format v2

--------EEPROM INFO--------
Vendor : StarFive Technology Co., Ltd.
Product full SN: VF7110B1-2253-D008E000-00004015
data version: 0x2
PCB revision: 0xb2
BOM revision: A
Ethernet MAC0 address: 6c:cf:39:00:42:ba
Ethernet MAC1 address: 6c:cf:39:00:42:bb
--------EEPROM INFO--------

In:    serial@10000000
Out:   serial@10000000
Err:   serial@10000000
Model: StarFive VisionFive V2
Net:   eth0: ethernet@16030000, eth1: ethernet@16040000
switch to partitions #0, OK
mmc1 is current device
found device 1
bootmode flash device 1
Can't set block device
Failed to load '/boot/uEnv.txt'
Hit any key to stop autoboot:  0
Can't set block device
Importing environment from mmc1 ...
## Warning: Input data exceeds 1048576 bytes - truncated
## Info: input data size = 1048578 = 0x100002
385 bytes read in 6 ms (62.5 KiB/s)
## Warning: defaulting to text format
## Error: "boot2" not defined
47546 bytes read in 13 ms (3.5 MiB/s)
47546 bytes written in 563 ms (82 KiB/s)
Retrieving file: /boot/extlinux/extlinux.conf
823 bytes read in 10 ms (80.1 KiB/s)
U-Boot menu
1:  Debian GNU/Linux bookworm/sid 5.15.0-starfive
2:  Debian GNU/Linux bookworm/sid 5.15.0-starfive (rescue target)
Enter choice: 1:    Debian GNU/Linux bookworm/sid 5.15.0-starfive
Retrieving file: /boot/initrd.img-5.15.0-starfive
9684953 bytes read in 421 ms (21.9 MiB/s)
Retrieving file: /boot/vmlinuz-5.15.0-starfive
8015200 bytes read in 348 ms (22 MiB/s)
append: root=/dev/mmcblk1p3 rw console=tty0 console=ttyS0,115200 earlycon rootwait stmmaceth=chain_mode:1 selinux=0
Retrieving file: /boot/dtbs/starfive/jh7110-visionfive-v2.dtb
47546 bytes read in 13 ms (3.5 MiB/s)
   Uncompressing Kernel Image
Moving Image from 0x44000000 to 0x40200000, end=41767000
## Flattened Device Tree blob at 48000000
   Booting using the fdt blob at 0x48000000
   Using Device Tree in place at 0000000048000000, end 000000004800e9b9

Starting kernel ...

clk u5_dw_i2c_clk_core already disabled
clk u5_dw_i2c_clk_apb already disabled
[    0.000000] Linux version 5.15.0-starfive (sw_buildbot@mdcsw02) (riscv64-unknown-linux-gnu-gcc (GCC) 10.2.0, GNU ld (GNU Binutils) 2.35) #1 SMP Mon Dec 19 07:56:37 EST 2022
[    0.000000] OF: fdt: Ignoring memory range 0x40000000 - 0x40200000
[    0.000000] Machine model: StarFive VisionFive V2
[    0.000000] earlycon: uart0 at MMIO32 0x0000000010000000 (options '115200')
[    0.000000] printk: bootconsole [uart0] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] Reserved memory: created CMA memory pool at 0x0000000080000000, size 512 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000040200000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x000000023fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000040200000-0x00000000c010ffff]
[    0.000000]   node   0: [mem 0x00000000c0110000-0x00000000c01fffff]
[    0.000000]   node   0: [mem 0x00000000c0200000-0x000000023fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000040200000-0x000000023fffffff]
[    0.000000] SBI specification v1.0 detected
[    0.000000] SBI implementation ID=0x1 Version=0x10002
[    0.000000] SBI TIME extension detected
[    0.000000] SBI IPI extension detected
[    0.000000] SBI RFENCE extension detected
[    0.000000] SBI SRST extension detected
[    0.000000] SBI v0.2 HSM extension detected
[    0.000000] CPU with hartid=0 is not available
[    0.000000] CPU with hartid=0 is not available
[    0.000000] riscv: ISA extensions acdfim
[    0.000000] riscv: ELF capabilities acdfim
[    0.000000] percpu: Embedded 17 pages/cpu s31528 r8192 d29912 u69632
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 2067975
[    0.000000] Kernel command line: root=/dev/mmcblk1p3 rw console=tty0 console=ttyS0,115200 earlycon rootwait stmmaceth=chain_mode:1 selinux=0
[    0.000000] Unknown command line parameters: stmmaceth=chain_mode:1 selinux=0
[    0.000000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, linear)
[    0.000000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: mapped [mem 0x00000000fbfff000-0x00000000fffff000] (64MB)
[    0.000000] Memory: 7582168K/8386560K available (9884K kernel code, 4982K rwdata, 4096K rodata, 2191K init, 401K bss, 280104K reserved, 524288K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=4.
[    0.000000] rcu:     RCU debug extended QS entry/exit.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] CPU with hartid=0 is not available
[    0.000000] riscv-intc: unable to find hart id for /cpus/cpu@0/interrupt-controller
[    0.000000] riscv-intc: 64 local interrupts mapped
[    0.000000] plic: plic@c000000: mapped 136 interrupts with 4 handlers for 9 contexts.
[    0.000000] random: get_random_bytes called from start_kernel+0x4d0/0x6e2 with crng_init=0
[    0.000000] riscv_timer_init_dt: Registering clocksource cpuid [0] hartid [1]
[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0x1d854df40, max_idle_ns: 881590404240 ns
[    0.000001] sched_clock: 64 bits at 4MHz, resolution 250ns, wraps every 2199023255500ns
[    0.009005] clocksource: timer@13050000.ch0: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635851949 ns
[    0.020330] clocksource: timer@13050000.ch1: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635851949 ns
[    0.031662] clocksource: timer@13050000.ch2: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635851949 ns
[    0.042988] clocksource: timer@13050000.ch3: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635851949 ns
[    0.054477] Console: colour dummy device 80x25
[    0.060704] printk: console [tty0] enabled
[    0.065259] Calibrating delay loop (skipped), value calculated using timer frequency.. 8.00 BogoMIPS (lpj=40000)
[    0.076482] pid_max: default: 32768 minimum: 301
[    0.081782] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.090298] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.100574] ASID allocator disabled
[    0.104515] rcu: Hierarchical SRCU implementation.
[    0.109965] EFI services will not be available.
[    0.115349] smp: Bringing up secondary CPUs ...
[    0.121954] smp: Brought up 1 node, 4 CPUs
[    0.128390] devtmpfs: initialized
[    0.139598] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.150460] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[    0.171957] pinctrl core: initialized pinctrl subsystem
[    0.178604] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.185824] cpuidle: using governor menu
[    0.212080] platform soc:dsi-output: Fixing up cyclic dependency with 29400000.dc8200
[    0.221266] platform 295d0000.mipi: Fixing up cyclic dependency with soc:dsi-output
[    0.230103] platform 29590000.hdmi: Fixing up cyclic dependency with 29400000.dc8200
[    0.250583] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.257968] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.268496] vgaarb: loaded
[    0.271719] SCSI subsystem initialized
[    0.276289] usbcore: registered new interface driver usbfs
[    0.282365] usbcore: registered new interface driver hub
[    0.288257] usbcore: registered new device driver usb
[    0.294100] mc: Linux media interface: v0.10
[    0.298849] videodev: Linux video capture interface: v2.00
[    0.305076] Advanced Linux Sound Architecture Driver Initialized.
[    0.312174] Bluetooth: Core ver 2.22
[    0.316137] NET: Registered PF_BLUETOOTH protocol family
[    0.321995] Bluetooth: HCI device and connection manager initialized
[    0.328993] Bluetooth: HCI socket layer initialized
[    0.334376] Bluetooth: L2CAP socket layer initialized
[    0.339947] Bluetooth: SCO socket layer initialized
[    0.345655] clocksource: Switched to clocksource riscv_clocksource
[    0.359945] NET: Registered PF_INET protocol family
[    0.366328] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.382150] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, linear)
[    0.391902] TCP established hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.401467] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, linear)
[    0.411645] TCP: Hash tables configured (established 65536 bind 65536)
[    0.419151] UDP hash table entries: 4096 (order: 6, 393216 bytes, linear)
[    0.427156] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, linear)
[    0.435832] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.442719] RPC: Registered named UNIX socket transport module.
[    0.449269] RPC: Registered udp transport module.
[    0.454444] RPC: Registered tcp transport module.
[    0.459629] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.467439] PCI: CLS 0 bytes, default 64
[    0.472432] Initialise system trusted keyrings
[    0.477573] workingset: timestamp_bits=62 max_order=21 bucket_order=0
[    0.477763] Unpacking initramfs...
[    0.490491] NFS: Registering the id_resolver key type
[    0.496101] Key type id_resolver registered
[    0.500717] Key type id_legacy registered
[    0.505234] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.512649] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[    0.520870] ntfs: driver 2.1.32 [Flags: R/W].
[    0.525964] jffs2: version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
[    0.533157] fuse: init (API version 7.34)
[    0.577419] NET: Registered PF_ALG protocol family
[    0.582730] Key type asymmetric registered
[    0.587294] Asymmetric key parser 'x509' registered
[    0.592771] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249)
[    0.600969] io scheduler mq-deadline registered
[    0.605983] io scheduler kyber registered
[    0.712788] clk-starfive-jh7110 13020000.clock-controller: starfive JH7110 clkgen init successfully.
[    0.724053] L2CACHE: DataError @ 0x00000000.08040140
[    0.729629] L2CACHE: DataFail @ 0x00000000.0804006C
[    0.735093] L2CACHE: No. of Banks in the cache: 8
[    0.740308] L2CACHE: No. of ways per bank: 16
[    0.745115] L2CACHE: Sets per bank: 256
[    0.749369] L2CACHE: Bytes per cache block: 64
[    0.754279] L2CACHE: Index of the largest way enabled: 15
[    0.760564] jh7110-pmu 17030000.power-controller: registered 8 power domains
[    0.828245] Serial: 8250/16550 driver, 6 ports, IRQ sharing disabled
[    0.838268] @@#########################@@
[    0.880540] @@ dev ptr:ffffffe0bff00000/1500/1
[    0.885723] PVR_K:  1: Read BVNC 36.50.54.182 from HW device registers
[    0.893003] PVR_K:  1: RGX Device registered BVNC 36.50.54.182 with 1 core in the system
[    0.903216] [drm] Initialized pvr 1.17.6210866 20170530 for 18000000.gpu on minor 0
[    0.924159] loop: module loaded
[    0.930154] spi-nor spi0.0: gd25lq128d (16384 Kbytes)
[    1.191214] Freeing initrd memory: 9456K
[    1.212281] 3 fixed-partitions partitions found on MTD device 13010000.spi.0
[    1.220111] Creating 3 MTD partitions on "13010000.spi.0":
[    1.226190] 0x000000000000-0x000000020000 : "spl"
[    1.233161] 0x000000100000-0x000000400000 : "uboot"
[    1.240125] 0x000000f00000-0x000001000000 : "data"
[    1.248093] libphy: Fixed MDIO Bus: probed
[    1.254081] CAN device driver interface
[    1.258941] starfive-eth-plat 16030000.ethernet: force_sf_dma_mode is ignored if force_thresh_dma_mode is set.
[    1.270362] starfive-eth-plat 16030000.ethernet: User ID: 0x41, Synopsys ID: 0x52
[    1.278643] starfive-eth-plat 16030000.ethernet:     DWMAC4/5
[    1.284695] starfive-eth-plat 16030000.ethernet: DMA HW capability register supported
[    1.293336] starfive-eth-plat 16030000.ethernet: RX Checksum Offload Engine supported
[    1.301978] starfive-eth-plat 16030000.ethernet: Wake-Up On Lan supported
[    1.309464] starfive-eth-plat 16030000.ethernet: TSO supported
[    1.315906] starfive-eth-plat 16030000.ethernet: Enable RX Mitigation via HW Watchdog Timer
[    1.325125] starfive-eth-plat 16030000.ethernet: Enabled Flow TC (entries=1)
[    1.332900] starfive-eth-plat 16030000.ethernet: TSO feature enabled
[    1.339913] starfive-eth-plat 16030000.ethernet: Using 40 bits DMA width
[    1.598015] libphy: stmmac: probed
[    1.601791] YT8531 Gigabit Ethernet stmmac-0:00: attached PHY driver (mii_bus:phy_addr=stmmac-0:00, irq=POLL)
[    1.612755] YT8531 Gigabit Ethernet stmmac-0:01: attached PHY driver (mii_bus:phy_addr=stmmac-0:01, irq=POLL)
[    1.624914] starfive-eth-plat 16040000.ethernet: force_sf_dma_mode is ignored if force_thresh_dma_mode is set.
[    1.636328] starfive-eth-plat 16040000.ethernet: User ID: 0x41, Synopsys ID: 0x52
[    1.644586] starfive-eth-plat 16040000.ethernet:     DWMAC4/5
[    1.650657] starfive-eth-plat 16040000.ethernet: DMA HW capability register supported
[    1.659300] starfive-eth-plat 16040000.ethernet: RX Checksum Offload Engine supported
[    1.667947] starfive-eth-plat 16040000.ethernet: Wake-Up On Lan supported
[    1.675420] starfive-eth-plat 16040000.ethernet: TSO supported
[    1.681863] starfive-eth-plat 16040000.ethernet: Enable RX Mitigation via HW Watchdog Timer
[    1.691113] starfive-eth-plat 16040000.ethernet: Enabled Flow TC (entries=1)
[    1.698891] starfive-eth-plat 16040000.ethernet: TSO feature enabled
[    1.705903] starfive-eth-plat 16040000.ethernet: Using 40 bits DMA width
[    1.962922] libphy: stmmac: probed
[    1.966737] YT8531 Gigabit Ethernet stmmac-1:00: attached PHY driver (mii_bus:phy_addr=stmmac-1:00, irq=POLL)
[    1.977684] YT8531 Gigabit Ethernet stmmac-1:01: attached PHY driver (mii_bus:phy_addr=stmmac-1:01, irq=POLL)
[    1.990177] Intel(R) Wireless WiFi driver for Linux
[    1.997530] cdns3-starfive 10210000.usbdrd: usb mode 2 2.0 probe success
[    2.005728] usbcore: registered new interface driver uas
[    2.011632] usbcore: registered new interface driver usb-storage
[    2.035993] starfive-rtc 17040000.rtc: registered as rtc0
[    2.041959] starfive-rtc 17040000.rtc: setting system clock to 2001-01-01T00:00:00 UTC (978307200)
[    2.052010] i2c_dev: i2c /dev entries driver
[    2.056986] usbcore: registered new interface driver uvcvideo
[    2.064494] starfive-wdt 13070000.wdog: Heartbeat: timeout=15, count/2=180000000 (0aba9500)
[    2.074249] Bluetooth: HCI UART driver ver 2.3
[    2.079178] Bluetooth: HCI UART protocol H4 registered
[    2.085056] starfive-cpufreq soc:starfive,jh7110-cpufreq: Failed to get regulator for cpu!
[    2.094201] starfive-cpufreq soc:starfive,jh7110-cpufreq: Failed to init starfive cpu dvfs info
[    2.104367] sdhci: Secure Digital Host Controller Interface driver
[    2.111207] sdhci: Copyright(c) Pierre Ossman
[    2.116050] Synopsys Designware Multimedia Card Interface Driver
[    2.122972] sdhci-pltfm: SDHCI platform and OF driver helper
[    2.130039] jh7110-sec 16000000.crypto: Unable to request sec_m dma channel in DMA channel
[    2.139189] jh7110-sec 16000000.crypto: Cannot initial dma chan
[    2.146014] usbcore: registered new interface driver usbhid
[    2.152166] usbhid: USB HID core driver
[    2.156657] usbcore: registered new interface driver snd-usb-audio
[    2.172080] NET: Registered PF_PACKET protocol family
[    2.177724] can: controller area network core
[    2.182629] NET: Registered PF_CAN protocol family
[    2.187939] can: raw protocol
[    2.191215] can: broadcast manager protocol
[    2.195856] can: netlink gateway - max_hops=1
[    2.200998] Bluetooth: RFCOMM TTY layer initialized
[    2.206424] Bluetooth: RFCOMM socket layer initialized
[    2.212118] Bluetooth: RFCOMM ver 1.11
[    2.216311] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[    2.222169] Bluetooth: BNEP filters: protocol multicast
[    2.227956] Bluetooth: BNEP socket layer initialized
[    2.233611] 9pnet: Installing 9P2000 support
[    2.238405] Key type dns_resolver registered
[    2.243868] Loading compiled-in X.509 certificates
[    2.285477] starfive_jh7110-pinctrl 13040000.gpio: SiFive GPIO chip registered 64 GPIOs
[    2.295094] starfive_jh7110-pinctrl 17020000.gpio: SiFive GPIO chip registered 4 GPIOs
[    2.304078] pl08xdmac 16008000.sec_dma: initialized 8 virtual memcpy channels
[    2.311992] pl08xdmac 16008000.sec_dma: initialized 16 virtual slave channels
[    2.321510] debugfs: Directory '16008000.sec_dma' with parent 'dmaengine' already present!
[    2.330687] pl08xdmac 16008000.sec_dma: DMA: PL080 rev0 at 0x16008000 irq 23
[    2.338771] ssp-pl022 10060000.spi: ARM PL022 driver for StarFive SoC platform, device ID: 0x00041022
[    2.348977] ssp-pl022 10060000.spi: mapped registers from 0x0000000010060000 to (____ptrval____)
[    2.359120] ssp-pl022 10060000.spi: Requested frequency: 10000000 Hz is unsupported,select by default 8250000 Hz
[    2.370741] ssp-pl022 10060000.spi: will use autosuspend for runtime pm, delay 100ms
[    2.380629] i2c 2-0045: Fixing up cyclic dependency with 295d0000.mipi
[    2.388122] seeed_panel 2-0045: Unknown Atmel firmware revision: 0x00
[    2.395431] i2c 2-0019: Fixing up cyclic dependency with 295d0000.mipi
[    2.404073] at24 5-0050: supply vcc not found, using dummy regulator
[    2.411894] at24 5-0050: 512 byte 24c04 EEPROM, writable, 16 bytes/write
[    2.421270] axp15060-regulator 5-0036: Register mipi_0p9 done! vol range:900 ~ 900 mV
[    2.431418] axp15060-regulator 5-0036: Register hdmi_1p8 done! vol range:1800 ~ 1800 mV
[    2.441765] axp15060-regulator 5-0036: Register hdmi_0p9 done! vol range:900 ~ 900 mV
[    2.451931] axp15060-regulator 5-0036: Register cpu_vdd done! vol range:500 ~ 1540 mV
[    2.461375] i2c 6-0010: Fixing up cyclic dependency with 19800000.vin_sysctl
[    2.469570] imx219 6-0010: supply VANA not found, using dummy regulator
[    2.477029] imx219 6-0010: supply VDIG not found, using dummy regulator
[    2.484377] imx219 6-0010: supply VDDL not found, using dummy regulator
[    2.499168] imx219 6-0010: failed to read chip id 219
[    2.504953] imx219: probe of 6-0010 failed with error -5
[    2.513417] pcie_plda 2b000000.pcie: host bridge /soc/pcie@2B000000 ranges:
[    2.521191] pcie_plda 2b000000.pcie:      MEM 0x0030000000..0x0037ffffff -> 0x0030000000
[    2.530144] pcie_plda 2b000000.pcie:      MEM 0x0900000000..0x093fffffff -> 0x0900000000
[    2.539130] ATR entry: 0x0940000000 -> 0x0000000000 [0x0010000000] (param: 0x000001)
[    2.547698] ATR entry: 0x0030000000 -> 0x0030000000 [0x0008000000] (param: 0x000000)
[    2.556265] ATR entry: 0x0900000000 -> 0x0900000000 [0x0040000000] (param: 0x000000)
[    2.905756] pcie_plda 2b000000.pcie: Port link up.
[    2.911202] pcie_plda 2b000000.pcie: PCI host bridge to bus 0000:00
[    2.918138] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.924191] pci_bus 0000:00: root bus resource [mem 0x30000000-0x37ffffff]
[    2.931793] pci_bus 0000:00: root bus resource [mem 0x900000000-0x93fffffff pref]
[    2.940084] pci 0000:00:00.0: [1556:1111] type 01 class 0x060400
[    2.946728] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0xffffffff 64bit pref]
[    2.954755] pci 0000:00:00.0: supports D1 D2
[    2.959479] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    2.970356] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.979343] pci 0000:01:00.0: [1106:3483] type 00 class 0x0c0330
[    2.986000] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00000fff 64bit]
[    2.993564] pci 0000:01:00.0: PME# supported from D0 D3cold
[    3.003108] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[    3.010448] pci 0000:00:00.0: BAR 0: no space for [mem size 0x100000000 64bit pref]
[    3.018918] pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x100000000 64bit pref]
[    3.027773] pci 0000:00:00.0: BAR 8: assigned [mem 0x30000000-0x300fffff]
[    3.035257] pci 0000:01:00.0: BAR 0: assigned [mem 0x30000000-0x30000fff 64bit]
[    3.043349] pci 0000:00:00.0: PCI bridge to [bus 01]
[    3.048836] pci 0000:00:00.0:   bridge window [mem 0x30000000-0x300fffff]
[    3.056432] pci 0000:00:00.0: enabling device (0000 -> 0002)
[    3.062673] pci 0000:01:00.0: enabling device (0000 -> 0002)
[    3.068956] pci 0000:01:00.0: quirk_usb_early_handoff+0x0/0x9d4 took 12239 usecs
[    3.077348] xhci_hcd 0000:01:00.0: xHCI Host Controller
[    3.083132] xhci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 1
[    3.091725] xhci_hcd 0000:01:00.0: hcc params 0x002841eb hci version 0x100 quirks 0x0000040000000890
[    3.101883] pcie_plda 2b000000.pcie: msi#0 address_hi 0x0 address_lo 0x190
[    3.109864] xhci_hcd 0000:01:00.0: xHCI Host Controller
[    3.115640] xhci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 2
[    3.123831] xhci_hcd 0000:01:00.0: Host supports USB 3.0 SuperSpeed
[    3.131508] hub 1-0:1.0: USB hub found
[    3.135704] hub 1-0:1.0: 1 port detected
[    3.141068] hub 2-0:1.0: USB hub found
[    3.145241] hub 2-0:1.0: 4 ports detected
[    3.152489] pcie_plda 2c000000.pcie: host bridge /soc/pcie@2C000000 ranges:
[    3.160247] pcie_plda 2c000000.pcie:      MEM 0x0038000000..0x003fffffff -> 0x0038000000
[    3.169208] pcie_plda 2c000000.pcie:      MEM 0x0980000000..0x09bfffffff -> 0x0980000000
[    3.178197] ATR entry: 0x09c0000000 -> 0x0000000000 [0x0010000000] (param: 0x000001)
[    3.186760] ATR entry: 0x0038000000 -> 0x0038000000 [0x0008000000] (param: 0x000000)
[    3.195309] ATR entry: 0x0980000000 -> 0x0980000000 [0x0040000000] (param: 0x000000)
[    3.355708] usb usb2-port2: over-current condition
[    3.455731] usb 1-1: new high-speed USB device number 2 using xhci_hcd
[    3.515736] usb usb2-port4: over-current condition
[    3.625932] pcie_plda 2c000000.pcie: Port link down, exit.
[    3.645690] clk-starfive-jh7110-vout 295c0000.clock-controller: starfive JH7110 clk_vout init successfully.
[    3.666898] hub 1-1:1.0: USB hub found
[    3.668071] clk-starfive-jh7110-isp 19810000.clock-controller: starfive JH7110 clk_isp init successfully.
done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... done.
Warning: fsck not present, so skipping root file system
[    7.114158] EXT4-fs (mmcblk1p3): mounted filesystem with ordered data mode. Opts: (null). Quota mode: disabled.
done.
Begin: Running /scripts/local-bottom ... done.
Begin: Running /scripts/init-bottom ... done.
[    7.883228] systemd[1]: System time before build time, advancing clock.
[    7.980379] systemd[1]: systemd 251.2-5 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    8.012496] systemd[1]: Detected architecture riscv64.

Welcome to D[    8.020576] systemd[1]: Hostname set to <starfive>.
ebian GNU/Linux bookworm/sid!

With that working it was time to try FreeBSD. A little searching unearthed a forums post showing a success opensbi->u-boot->loader process, but almost immediate failure once the kernel started. The forums post used a stock 13 image to boot which made it look like getting to a partially booting kernel wouldn't require too much extra work even if there was an immediate boot issue to fix.

The Debian boot process initially recommended manually picking up the kernel and device tree images to boot. A little googling pulled up a git repo with a script and instructions on how to boot . With that magic I grabbed a current snapshot and tried to reproduce the failing boot.

U-Boot SPL 2021.10 (Feb 12 2023 - 18:15:33 +0800)
DDR version: dc2e84f0.
Trying to boot from SPI

OpenSBI v1.2
   ____                    _____ ____ _____
  / __ \                  / ____|  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) |  __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|____/_____|
        | |
        |_|

Platform Name             : StarFive VisionFive V2
Platform Features         : medeleg
Platform HART Count       : 5
Platform IPI Device       : aclint-mswi
Platform Timer Device     : aclint-mtimer @ 4000000Hz
Platform Console Device   : uart8250
Platform HSM Device       : jh7110-hsm
Platform PMU Device       : ---
Platform Reboot Device    : pm-reset
Platform Shutdown Device  : pm-reset
Firmware Base             : 0x40000000
Firmware Size             : 292 KB
Runtime SBI Version       : 1.0

Domain0 Name              : root
Domain0 Boot HART         : 1
Domain0 HARTs             : 0*,1*,2*,3*,4*
Domain0 Region00          : 0x0000000002000000-0x000000000200ffff (I)
Domain0 Region01          : 0x0000000040000000-0x000000004007ffff ()
Domain0 Region02          : 0x0000000000000000-0xffffffffffffffff (R,W,X)
Loading kernel...
/boot/kernel/kernel text=0x5c324c text=0x182fa4 data=0xf5508 data=0x2dc4+0x264a4c 0x8+0x1ffc5c0+0x8+0xfb1fc/
Loading configured modules...
/boot/kernel/umodem.ko text=0x2080 text=0x1280 data=0x6f0+0x4 0x8+0x71b8+0x8+0xdcc
loading required module 'ucom'
/boot/kernel/ucom.ko text=0x2618 text=0x2d28 data=0x960+0x858 0x8+0xfae0+0x8+0x1492
can't find '/etc/hostid'
can't find '/boot/entropy'

Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel]...               
Using DTB provided by EFI at 0x47ef2000.
Kernel entry at 0xf680002e...
Kernel args: (null)
clk u5_dw_i2c_clk_core already disabled
clk u5_dw_i2c_clk_apb already disabled
---<<BOOT>>---
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.0-CURRENT #0 main-n263139-baef3a5b585f: Thu May 25 06:56:04 UTC 2023
    root@releng1.nyi.freebsd.org:/usr/obj/usr/src/riscv.riscv64/sys/GENERIC riscv
FreeBSD clang version 15.0.7 (https://github.com/llvm/llvm-project.git llvmorg-15.0.7-0-g8dfdcc7b7bf6)
WARNING: WITNESS option enabled, expect reduced performance.
VT: init without driver.
SBI: OpenSBI v1.2
SBI Specification Version: 1.0
CPU 0  : Vendor=SiFive Core=6/7/P200/X200-Series Processor (Hart 1)
  marchid=0x8000000000000007, mimpid=0x4210427
  MMU: 0x1<Sv39>
  ISA: 0x112d<Atomic,Compressed,Double,Float,Mult/Div>
real memory  = 4294967296 (4096 MB)
avail memory = 4122566656 (3931 MB)
sbi_trap_error: hart0: trap handler failed (error -2)
sbi_trap_error: hart0: mcause=0x0000000000000005 mtval=0x0000000040048060
sbi_trap_error: hart0: mepc=0x0000000040004cac mstatus=0x0000000200001800
sbi_trap_error: hart0: ra=0x0000000040009ee2 sp=0x0000000040047f10
sbi_trap_error: hart0: gp=0x0000000000000000 tp=0x0000000040048000
sbi_trap_error: hart0: s0=0x0000000040047f20 s1=0x0000000040048000
sbi_trap_error: hart0: a0=0x0000000040048060 a1=0x0000000000000002
sbi_trap_error: hart0: a2=0x0000000000000000 a3=0x0000000000000019
sbi_trap_error: hart0: a4=0x0000000000000001 a5=0x0000000040048060
sbi_trap_error: hart0: a6=0x00000000400480a8 a7=0x0000000000000004
sbi_trap_error: hart0: s2=0x00000000400241a8 s3=0x0000000000000000
sbi_trap_error: hart0: s4=0x0000000000000000 s5=0x0000000040029000
sbi_trap_error: hart0: s6=0x0000000040029020 s7=0x0000000000000000
sbi_trap_error: hart0: s8=0x000000000000001c s9=0x0000000040035ab0
sbi_trap_error: hart0: s10=0x0000000000000000 s11=0x0000000000000000
sbi_trap_error: hart0: t0=0x0000000000000000 t1=0x0000000000000000
sbi_trap_error: hart0: t2=0x0000000000000000 t3=0x0000000000002000
sbi_trap_error: hart0: t4=0x0000000000000000 t5=0x0000000000000000
sbi_trap_error: hart0: t6=0x0000000000000000

Cool, I can reproduce the failure. I love it when that happens.

The git repo I got the u-boot commands from was adding a memory disk image to the boot process and showed a system log of the host getting all the way up to single user (without any device specific devices discovered). Digging into their script they pull down a different device tree blob and use that with their manual u-boot boot.

There was a follow up post by Jess (jrtc27@) linking to a issue with the device tree blob . I guess her suggestions were merged into the visionfive2 tree and they have yet to make it through Linux and into FreeBSD.

We need an updated DTB.

Loading kernel...
/boot/kernel/kernel text=0x5c324c text=0x182fa4 data=0xf5508 data=0x2dc4+0x264a4c 0x8+0x1ffc5c0+0x8+0xfb1fc/
Loading configured modules...
can't find '/boot/entropy'
/boot/kernel/umodem.ko text=0x2080 text=0x1280 data=0x6f0+0x4 0x8+0x71b8+0x8+0xdcc
loading required module 'ucom'
/boot/kernel/ucom.ko text=0x2618 text=0x2d28 data=0x960+0x858 0x8+0xfae0+0x8+0x1492
can't find '/etc/hostid'

Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel]...               
Using DTB provided by EFI at 0x47f00000.
Kernel entry at 0xf680002e...
Kernel args: (null)
clk u5_dw_i2c_clk_core already disabled
clk u5_dw_i2c_clk_apb already disabled
---<<BOOT>>---
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.0-CURRENT #0 main-n263139-baef3a5b585f: Thu May 25 06:56:04 UTC 2023
    root@releng1.nyi.freebsd.org:/usr/obj/usr/src/riscv.riscv64/sys/GENERIC riscv
FreeBSD clang version 15.0.7 (https://github.com/llvm/llvm-project.git llvmorg-15.0.7-0-g8dfdcc7b7bf6)
WARNING: WITNESS option enabled, expect reduced performance.
VT: init without driver.
SBI: OpenSBI v1.2
SBI Specification Version: 1.0
CPU 0  : Vendor=SiFive Core=6/7/P200/X200-Series Processor (Hart 1)
  marchid=0x8000000000000007, mimpid=0x4210427
  MMU: 0x1<Sv39>
  ISA: 0x112d<Atomic,Compressed,Double,Float,Mult/Div>
real memory  = 4294967296 (4096 MB)
avail memory = 4121706496 (3930 MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
CPU 1  : Vendor=SiFive Core=6/7/P200/X200-Series Processor (Hart 2)
CPU 2  : Vendor=SiFive Core=6/7/P200/X200-Series Processor (Hart 3)
CPU 3  : Vendor=SiFive Core=6/7/P200/X200-Series Processor (Hart 4)
arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled.
random: entropy device external interface
kbd0 at kbdmux0
ofwbus0: <Open Firmware Device Tree>
clk_fixed0: <Fixed clock> on ofwbus0
clk_fixed1: <Fixed clock> on ofwbus0
clk_fixed2: <Fixed clock> on ofwbus0
clk_fixed3: <Fixed clock> on ofwbus0
clk_fixed4: <Fixed clock> on ofwbus0
clk_fixed5: <Fixed clock> on ofwbus0
clk_fixed6: <Fixed clock> on ofwbus0
clk_fixed7: <Fixed clock> on ofwbus0
clk_fixed8: <Fixed clock> on ofwbus0
clk_fixed9: <Fixed clock> on ofwbus0
clk_fixed10: <Fixed clock> on ofwbus0
clk_fixed11: <Fixed clock> on ofwbus0
clk_fixed12: <Fixed clock> on ofwbus0
clk_fixed13: <Fixed clock> on ofwbus0
clk_fixed14: <Fixed clock> on ofwbus0
clk_fixed15: <Fixed clock> on ofwbus0
clk_fixed16: <Fixed clock> on ofwbus0
clk_fixed17: <Fixed clock> on ofwbus0
clk_fixed18: <Fixed clock> on ofwbus0
clk_fixed19: <Fixed clock> on ofwbus0
simplebus0: <Flattened device tree simple bus> on ofwbus0
plic0: <RISC-V PLIC> mem 0xc000000-0xfffffff irq 14,15,16,17,18,19,20,21,22 on simplebus0
timer0: <RISC-V Timer>
Timecounter "RISC-V Timecounter" frequency 4000000 Hz quality 1000
Event timer "RISC-V Eventtimer" frequency 4000000 Hz quality 1000
rcons0: <RISC-V console>
cpulist0: <Open Firmware CPU Group> on ofwbus0
cpu0: <Open Firmware CPU> on cpulist0
Timecounters tick every 1.000 msec
usb_needs_explore_all: no devclass
Release APs
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/ufs/rootfs [rw]...
mountroot: waiting for device /dev/ufs/rootfs...
random: unblocking device.
Mounting from ufs:/dev/ufs/rootfs failed with error 19.

Loader variables:
  vfs.root.mountfrom=ufs:/dev/ufs/rootfs
  vfs.root.mountfrom.options=rw

Manual root filesystem specification:
  <fstype>:<device> [options]
      Mount <device> using filesystem <fstype>
      and with the specified (optional) option list.

    eg. ufs:/dev/da0s1a
    zfs:zroot/ROOT/default
    cd9660:/dev/cd0 ro
      (which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /)

  ?               List valid disk boot devices
  .               Yield 1 second (for background tasks)
  <empty line>    Abort manual input

mountroot>

Cool that works, but it is a very annoying way to boot. Can we fix it?

StarFive # printenv
baudrate=115200
boot_a_script=load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script}; source ${scriptaddr}
boot_efi_binary=load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} efi/boot/bootriscv64.efi; if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r};else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi
boot_efi_bootmgr=if fdt addr ${fdt_addr_r}; then bootefi bootmgr ${fdt_addr_r};else bootefi bootmgr;fi
boot_extlinux=sysboot ${devtype} ${devnum}:${distro_bootpart} any ${scriptaddr} ${prefix}${boot_syslinux_conf}
boot_prefixes=/ /boot/
boot_script_dhcp=boot.scr.uimg
boot_scripts=boot.scr.uimg boot.scr
boot_syslinux_conf=extlinux/extlinux.conf
boot_targets=mmc0 dhcp
bootargs=console=ttyS0,115200  debug rootwait  earlycon=sbi
bootcmd=run load_vf2_env;run importbootenv;run load_distro_uenv;run boot2;run distro_bootcmd
bootcmd_dhcp=devtype=dhcp; if dhcp ${scriptaddr} ${boot_script_dhcp}; then source ${scriptaddr}; fi;setenv efi_fdtfile ${fdtfile}; setenv efi_old_vci ${bootp_vci};setenv efi_old_arch ${bootp_arch};setenv bootp_vci PXEClient:Arch:00027:UNDI:003000;setenv bootp_arch 0x1b;if dhcp ${kernel_addr_r}; then tftpboot ${fdt_addr_r} dtb/${efi_fdtfile};if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r}; else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi;fi;setenv bootp_vci ${efi_old_vci};setenv bootp_arch ${efi_old_arch};setenv efi_fdtfile;setenv efi_old_arch;setenv efi_old_vci;
bootcmd_distro=run fdt_loaddtb; run fdt_sizecheck; run set_fdt_distro; sysboot mmc ${fatbootpart} fat c0000000 ${bootdir}/${boot_syslinux_conf};
bootcmd_mmc0=devnum=0; run mmc_boot
bootdelay=2
bootdir=/boot
bootenv=uEnv.txt
bootmode=flash
bootpart=1:3
chip_vision=B
chipa_gmac_set=fdt set /soc/ethernet@16030000/ethernet-phy@0 tx_inverted_10 <0x0>;fdt set /soc/ethernet@16030000/ethernet-phy@0 tx_inverted_100 <0x0>;fdt set /soc/ethernet@16030000/ethernet-phy@0 tx_inverted_1000 <0x0>;fdt set /soc/ethernet@16030000/ethernet-phy@0 tx_delay_sel <0x9>;fdt set /soc/ethernet@16040000/ethernet-phy@1 tx_inverted_10 <0x0>;fdt set /soc/ethernet@16040000/ethernet-phy@1 tx_inverted_100 <0x0>;fdt set /soc/ethernet@16040000/ethernet-phy@1 tx_inverted_1000 <0x0>;fdt set /soc/ethernet@16040000/ethernet-phy@1 tx_delay_sel <0x9>
chipa_set=if test ${chip_vision} = A; then run chipa_gmac_set;fi;
chipa_set_linux=fdt addr ${fdt_addr_r};run visionfive2_mem_set;run chipa_set;
chipa_set_linux_force=fdt addr ${fdt_addr_r};run visionfive2_mem_set;run chipa_gmac_set;
chipa_set_uboot=fdt addr ${uboot_fdt_addr};run chipa_set;
chipa_set_uboot_force=fdt addr ${uboot_fdt_addr};run chipa_gmac_set;
devnum=1
distro_bootcmd=for target in ${boot_targets}; do run bootcmd_${target}; done
distroloadaddr=0xb0000000
efi_dtb_prefixes=/ /dtb/ /dtb/current/
eth0addr=6c:cf:39:00:42:ba
eth1addr=6c:cf:39:00:42:bb
ethaddr=6c:cf:39:00:42:ba
ext4bootenv=ext4load mmc ${bootpart} ${loadaddr} ${bootdir}/${bootenv}
fatbootpart=1:2
fdt_addr_r=0x46000000
fdt_high=0xffffffffffffffff
fdt_loaddtb=fatload mmc ${fatbootpart} ${fdt_addr_r} ${bootdir}/dtbs/${fdtfile}; fdt addr ${fdt_addr_r};
fdt_sizecheck=fatsize mmc ${fatbootpart} ${bootdir}/dtbs/${fdtfile};
fdtaddr=fffc5dd0
fdtcontroladdr=fffc5dd0
fdtfile=starfive/starfive_visionfive2.dtb
importbootenv=echo Importing environment from mmc${devnum} ...; env import -t ${loadaddr} ${filesize}
initrd_high=0xffffffffffffffff
ipaddr=192.168.120.230
kernel_addr_r=0x40200000
load_distro_uenv=fatload mmc ${fatbootpart} ${distroloadaddr} ${bootdir}/${bootenv}; env import ${distroloadaddr} 17c;
load_efi_dtb=load ${devtype} ${devnum}:${distro_bootpart} ${fdt_addr_r} ${prefix}${efi_fdtfile}
load_vf2_env=fatload mmc ${bootpart} ${loadaddr} ${testenv}
loadaddr=0xa0000000
loadbootenv=fatload mmc ${bootpart} ${loadaddr} ${bootenv}
memory_addr=40000000
memory_size=200000000
mmc_boot=if mmc dev ${devnum}; then devtype=mmc; run scan_dev_for_boot_part; fi
mmcbootenv=run scan_mmc_dev; setenv bootpart ${devnum}:${mmcpart}; if mmc rescan; then run loadbootenv && run importbootenv; run ext4bootenv && run importbootenv; if test -n $uenvcmd; then echo Running uenvcmd ...; run uenvcmd; fi; fi
mmcpart=3
netmask=255.255.255.0
partitions=name=loader1,start=17K,size=1M,type=${type_guid_gpt_loader1};name=loader2,size=4MB,type=${type_guid_gpt_loader2};name=system,size=-,bootable,type=${type_guid_gpt_system};
preboot=run chipa_set_uboot;run mmcbootenv
pxefile_addr_r=0x45900000
ramdisk_addr_r=0x46100000
scan_dev_for_boot=echo Scanning ${devtype} ${devnum}:${distro_bootpart}...; for prefix in ${boot_prefixes}; do run scan_dev_for_extlinux; run scan_dev_for_scripts; done;run scan_dev_for_efi;
scan_dev_for_boot_part=part list ${devtype} ${devnum} -bootable devplist; env exists devplist || setenv devplist 1; for distro_bootpart in ${devplist}; do if fstype ${devtype} ${devnum}:${distro_bootpart} bootfstype; then run scan_dev_for_boot; fi; done; setenv devplist
scan_dev_for_efi=setenv efi_fdtfile ${fdtfile}; for prefix in ${efi_dtb_prefixes}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${efi_fdtfile}; then run load_efi_dtb; fi;done;run boot_efi_bootmgr;if test -e ${devtype} ${devnum}:${distro_bootpart} efi/boot/bootriscv64.efi; then echo Found EFI removable media binary efi/boot/bootriscv64.efi; run boot_efi_binary; echo EFI LOAD FAILED: continuing...; fi; setenv efi_fdtfile
scan_dev_for_extlinux=if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${boot_syslinux_conf}; then echo Found ${prefix}${boot_syslinux_conf}; run boot_extlinux; echo SCRIPT FAILED: continuing...; fi
scan_dev_for_scripts=for script in ${boot_scripts}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${script}; then echo Found U-Boot script ${prefix}${script}; run boot_a_script; echo SCRIPT FAILED: continuing...; fi; done
scan_mmc_dev=if test ${bootmode} = flash; then if mmc dev ${devnum}; then echo found device ${devnum};else setenv devnum 0;mmc dev 0;fi; fi; echo bootmode ${bootmode} device ${devnum};
scan_sf_for_scripts=${devtype} read ${scriptaddr} ${script_offset_f} ${script_size_f}; source ${scriptaddr}; echo SCRIPT FAILED: continuing...
script_offset_f=0x1fff000
script_size_f=0x1000
scriptaddr=0x43900000
serial#=VF7110B1-2253-D008E000-00004015
set_fdt_distro=if test ${chip_vision} = A; then if test ${memory_size} = 200000000; then run chipa_gmac_set;run visionfive2_mem_set;fatwrite mmc ${fatbootpart} ${fdt_addr_r} ${bootdir}/dtbs/${fdtfile} ${filesize};else run chipa_gmac_set;run visionfive2_mem_set;fatwrite mmc ${fatbootpart} ${fdt_addr_r} ${bootdir}/dtbs/${fdtfile} ${filesize};fi;else run visionfive2_mem_set;fatwrite mmc ${fatbootpart} ${fdt_addr_r} ${bootdir}/dtbs/${fdtfile} ${filesize};fi;
sf_boot=if sf probe ${busnum}; then devtype=sf; run scan_sf_for_scripts; fi
stderr=serial@10000000
stdin=serial@10000000
stdout=serial@10000000
testenv=vf2_uEnv.txt
type_guid_gpt_loader1=5B193300-FC78-40CD-8002-E86C45580B47
type_guid_gpt_loader2=2E54B353-1271-4842-806F-E436D6AF6985
type_guid_gpt_system=0FC63DAF-8483-4772-8E79-3D69D8477DE4
uboot_fdt_addr=0xfffc5dd0
ver=U-Boot 2021.10 (Feb 12 2023 - 18:15:33 +0800)
visionfive2_mem_set=fdt memory ${memory_addr} ${memory_size};

Environment size: 7216/65532 bytes

From the u-boot environment we can find the DTB file u-boot is passing on to the EFI loader. If we move our DTB to be in that location on disk we should be able to boot automatically.

fdtfile=starfive/starfive_visionfive2.dtb

With the working DTB in the correct place we can boot the FreeBSD image from power on without any manual intervention. We fail at mount root because we are missing drivers for any sort of storage medium, eMMC or SD (both are on a SDIO controller), USB or NVMe. We don't have any drivers for this board yet, the stuff detected in the dmesg above is devices in the DTB which exist elsewhere and so have FreeBSD drivers.

To get past mount root (without using a memory disk) we need to have a storage driver, so lets pull up the docs and write one!

Much angered reading and intense searching later

It looks like there isn't useful documentation available for a driver writer. There are open source drivers on their way into the Linux Kernel, but the Software TRM is a high level overview, the software developer guides explain how to use interfaces from Linux and everything else is a useless summary.

From searching I found that Haiku is booting and running on the VisionFive2 . They have drivers for the PCIe controller giving them USB and NVMe and graphics on the frame buffer because u-boot is kind enough to set it up. As far as I can tell (I hope I'm wrong), they PCIe controller driver has been cribbed from the Linux Kernel.

While writing drivers from others source is possible, it is a bit hairy going from the GPL to BSD licenses and it is terribly annoying when you hit issues to not have documentation.

I can't see FreeBSD supporting the VisionFive2 without there being proper open documentation to write drivers from.

Some ideas I picked from the electromagnetic fields

I made it to and from the Electromagnetic Field . It was quite an experience to return to a hacker large event after so many years.

A real review of the event will appear in the FreeBSD Journal this summer and I rambled a little at the end of BSDNow 459.

I don't think I could have articulated before hand exactly how much I missed the massive influx of energy being around people doing things gives you. To be honest, it probably explains why I spent so many years running things myself. The sheer magic of other people doing things seems to relax my brain and lets it fill up to the brim with new ideas.

In the vein of "“Make 50 of Something" ( which I learned about from Vi Hart making 50 fizzbuzzers ), I thought it was finally time to write down the ideas in the aftermath of the event and have a record of them.

50 is a lot of things, but I don't think this list actually captures everything that popped into my head. Much of this list is ideas I have been having over the last decade of going to hacker events, but unexecuted ideas can still be good.

  1. ID Badge
  2. Buckfast FM
  3. Buckfast FM badge
  4. BBS serial network
  5. Government style news distribution network with backdoor leading to a bbs
  6. TIC-80 simulator for the badge
  7. shit camera ID printer
  8. badge tamogotchi thing
  9. "Cyberman" led outfit
  10. kit dome of trees
  11. web bbs
  12. retro bbs
  13. web bbs serial links to esp consoles
  14. demo display on a dome
  15. longest serial transmission competition
  16. internet of buckets
  17. network of floaty led things in lake
  18. posters!
  19. stickers!
  20. sticker: The Internet is Bullshit
  21. sticker: Computers are a fuck
  22. sticker: "I'll drive if you feed me cheese"
  23. sticker: a big pile of tofu
  24. talk: 10 years of Scotland as performance art
  25. Old Unix BBS
  26. something targetting the badge
  27. "set" mainframe backdrop
  28. weather reports
  29. Scotcon TV
  30. An actual good computer book shop
  31. BSD meet up
  32. hacker breakfast
  33. 'drugwars' with real location integration
  34. CTF coffee puzzle
  35. folding bikes!
  36. portraits with a camera
  37. portraits with a paint brush
  38. computer church
  39. beacon hunt
  40. demo party
  41. paiting party
  42. talk: Learning a place by drawing its buildings
  43. SBC colo
  44. deerocracy
  45. tshirts
  46. sticker: No one hsa found a good use for a networked computer
  47. sketching workshop
  48. thicknet, network
  49. industrial quantities of mauve
  50. $5 electronics macro photo workshop
  51. silent journalling with [tj]

Yes, It is super vague. There are thousands of words of explanation behind some of these. That you will have to invent for yourself.

Steal what you want, I won't remember publishing this and will love seeing your implementation.

next